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ABSTRACT 

This report covers the Phase I activities of a 
project for the development of Instructional Systems Development 
(ISD) based training . programs for Marine and Navy F-U pilots and 
Radar Intercept Officers (RIO) . In addition to developing specified 
end-pro.ducts to be used in subsequent ISD phases, it was an effort to 
evaluate the utility of the emerging 6.2 ISD model by identifying its 
strength53 and weaknesses. Project activities were conducted on-site 
at the fiarine Corps Air Station, Yuma, with government furnished 
subject matter experts providing input to major tasks under the 
direction of Allen Corporation ISD personnel. Major tasks completed 
during this project were objectives hierarchy development, media 
selection, syllabus development, and training support requirements 
analysis. Each of these activities is summarized. (Author/JEG) 
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This project was one in a continuing series of studies to upgrade 
Naval aviation aircrew training prcgrams and at the same time to study 
the methodology, effectiveness, ano resource requirements of Instructional 
Systems Development (ISD). It was a Phase I effort encompassing ISD steps 
from hierarchy development through training support requirements analysis , 
Project goals were to support the Marine and Navy Fleet Readiness Squadrons 
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in their training program development efforts and to study the utility 
of the 6.2 ISD model in an application to an existing aviation system. 
The latter goal included gathering information on the strengths and 
weaknesses of the model, making recommendations for modifications to 
the model, and collecting resource utilization data. Input data came 
from a validated list of pilot and RIO tasks which was furnished by 
the government. Tasks from this list were analyzed into a hierarchical 
structure of supporting skills and knowledge. This analysis formed 
the basis for the development of instructional objectives. Preferred 
and alternate media were selected for each of the objectives. Objectives 
were then grouped into lessons and the lessons sequenced to form the 
pilot and RIO syllabi. Finally, a training support requirements analysis 
was performed to estimate resource requirements for development, imple- 
mentation, revisiori^^ and maintenance of the two training courses. 
Problems encountered during this program were discussed and recommen- 
dations for changes to the ISD model were presented. 
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SUMMARY 

This report covers the Phase I activities of a project 
for development ISD-based training programs for Marine and 
Navy F-4 pilots anc Radar Intercept Officers (RIO). In addition 
to developinc ^^ecified end-products to be used in subsequent 
ISD phases, ir -as an effort to evaluate the utility of the 
emerging 6-2 ISD model by identifying its strengths and weak- 
nesses. Project activities were conducted on-site at MCAS , Yuma 
with government-furnished subject matter experts providing input 
to major tasks under the direction of Allen Corporation ISD 
personnel . 

Major tasks completed during this project were objectives 
hierarchy development, media selection, syllabus development, and 
training s.^-port requirements analysis. Each of these activities 
is summarized below. 

OBJECTIVES HIERARCHY DEVELOPMENT 

A job analysis was conducted prior to contract award to 
Allen Corporation. This survey provided a listing of tasks 
selected for training. These tasks were the input to objectives 
hierarchy development. During hierarchy development, tasks were 
analyzed to identify subtasks which were components of the major 
tasks. Behavioral objectives were developed for each subtask 
and, along with the major tasks, were arranged in hierarchy 
formats to show their interrelationships, i.e. superordi nate, 
coordinate, and subordinate relationships - 

MEDIA SELECTION 

Preferred and alternate instructional media were selected 
for each objective identified during hierarchy development. 
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Selections were made on the basis of an algorithm in which 
the following variables were considered: (1) level of expected 
student behavior, (2) Vevel of content taught, (3) amount of 
practice required for mastery, (4) minimum display requirements, 
and (5) amount of memorization required. During the next step, 
syllabus development, media selections for the objectives 
were used to determine the preferred medium for each lesson. 

SYLLABUS DEVELOPMENT 

Objectives were grouped to forn^ lessons. First, flight 
lessons were formulated using objectives selected for flights 
during m^dia selection. Next, objectives requiring simulators 
were grouped to form lessons to support the previously formed 
flight lessons. Finally, ground school lessons were structured 
to support the simulator and flight lessons. After the ground 
school objectives were grouped into lessons, media selections 
were reviewed to determine if media for all objectives in a 
lesson were the same. If not, adjustments in media and/or 
lesson content were made to select the preferred medium for 
each lesson. 

The resultant syllabus embodied an individualized instruction- 
al system in which the predominant medium was slide-tape presen- 
tation. Other media included workbooks, videotapes, simulators, 
and the aircraft. Lessons were sequenced to f^.cilitate acquisition 
and retention by proceeding from ground school instruction to 
hands-on practice in a simulator, to practice in the aircraft. 

TRAINING SUPPORT REQUIREMENTS ANALYSIS 

The final task of this project was to estimate cost, 
facilities, and personnel requirements to develop, implement, 
revise and maintain the training program outlined in the 
syllabus. Development cost estimates were based on a survey of 
coiTimercial and government organizations. This survey yielded 
costs and personnel requirements for lessons in each of the 
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selected media. Similar requirements for revision of materials 
were also determined. Facilities requirements included that 
required by materials development personnel and for implementation, 
of the training course, 

CONCLUSIONS 

The general conclusion of this project was that the ISO 
model used provides a viable approach to ISD implementation. 
A number of problems were encountered, which resulted in suggestions 
for modifications to the model. The experience gained during this 
project and the resultant recommendations will assist in the 
further refinement of the 6.2 model- 



NAVTRAEQUIPCEN 77-C-0081-1 



PREFACE 



The Naval Training Equipment Center has a continuing 
interest in the use of ISD techniques to upgrade aircrew 
training programs and in the process, to develop and refine 
a model for future ISD programs. This F-4 program followed 
four similar ISD programs (SH-2F LAMPS, A-6E TRAM, EA-6B ICAP, 
and E-2C HAWKEYE) in which different ISD methodologies were 
used by the "contractors involved. As a result of these pro- 
grams, a standard ISD model was developed. The F-4 program is 
the first program in which the model has been implemented by . 
a contractor other than those involved in the previous ISD 
programs. 'The work was performed by the Allen Corporation 
of America under Contract N61339-7^-C-0081 , A multi-phase 
ISD effort was planned- The Phase One analys.is was completed 
and is described in this report. 

The operational goal of this project was to develop the 
framework for pilot and RIO training programs, which coulc 
be expanded into instructional materials for implementation 
in subsequent phases. The research and development goals 
were to: (a) evaluate how well an independent contractor, using 
the detailed ISD specification and associated Data Iter: 
Descriptions, could implement the model, (b) identify strengths 
and weaknesses of the model, (c) obtain recommendations for 
modifications to the model and (d) acquire cost, scheduling, 
and manpower data for future ISD planning. 




WILLIAM B, BONEY 
Scientific Officer 
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SECTION I 
INTRODUCTION 

BACKGROUND 

This project is the latest in a series of programs sponsored 
by the Naval Training Equipment Center (NAVTRAEQUI PCEN) to develop, 
refine, and standardize techniques used in Instructional Systems 
Development (ISD). The goal of these programs is to define an ISO 
model which will provide a standardized approach to developing 
training programs for operators and maintenance personnel and at 
the same time to upgrade the quality of aircrew training on 
selector aviation systems. 

The first four ISD programs in the series were carried ou\^ 
simultaneously by different contractors under broad ISD guidelines 
'specified by NAVTRAEQUIPCEN . The purpose of providing only broad 
guidelines for discrete processes and end products was to allow 
the different contractors flexibility to utilize their own tech- 
niques and- procedures. Under the four contracts the analysis - 
steps in the ISD process up through lesson specifications were 
carried out for four different aircrew training courses. The 
aircrewmen for which the ISD process was initiated were the pilot, 
NFOs, and FT of the E-2C Hawkeye, the pilot and ECMOs of the 
EA-6B Prowler, the pilot and sensor operator of the SH-2F LAMPS, 
and the pilot and NFO of the A-6E TRAM, 

Each of the four programs required the development of 
training programs for two different types of crewmembers: (1) 
pilot training which has a large manipulative skills component 
supplemented with cogni.tive instruction; and (2) NFO training, 
which primarily -invQl ves instruction in interpretation and decision- 
making processes. The programs also encompassed a wide range of 
aircraft types including an attack aircraft (A-6E), an electronic 
countermeasures aircraft (EA-6B), an airborne early warning and 
surveillance aircraft (E-2C), and an anti -submari ne helicopter 
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(SH-2F). These four programs with their variety of aircrew types 
and aircraft missions provided the information from which a basic 
Navy model for ISD was developed. In addition, a large amount of 
data on the manpower and costs required to do large ISD programs 
was collected. 

The second step in the series of ISD programs v;as the 
formulation of a model using as a data base the procedures employed 
by the different contractors in the four previous programs. The 
purpose was to analyze the strengths and weaknesses of the different 
approaches to ISD and to synthesize the strengths of the models into 
one model which would become the Navy standard for training program 
development. The use of the model in future programs would facilitate 
management by contract technical monitors and would insure a uniformly 
high quality product. The resultant model has been referred to as 
the "6.2 Model". The most recent version of the model is delineated- 
in Military Specification MIL-T-29053» dated 19 October 1977, and 
is entitled "Training Requirements for Aviation Weapon Systems". 

The F-4 program which is the most recent in the series of 
ISD programs, emerged as the result of two needs. One was the 
recognized shortcomings of the F-4 pilot and Radar Intercept 
.Officer (RIO) training courses. The second was the need to evaluate 
how effectively a contractor, not involved in the earlier ISD 
programs, could apply the "6.2 Model" to an existing aircrew 
training program. Related to this second need was the continuing 
requirement to collect data on ISD manpower and cost requirements. 

Specific deficiencies in the two F-4 training courses were 
identified in the problem analysis. It was noted that objective 
performance standards and standardized criteria and evaluation 
methods have not been defined for academic, simulator, or flight 
training. This lack of standardization and specific evaluation 
criteria significantly inhibits efforts to improve the training. 
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Furthermore, an extremely low wash-out rate suggests that the exit 
level skills of the trainees are not being assessed with adequate 
precision. Hence, optimal remedial action, appropriate to students 
and/or the training program is hampered. It was also noted that 
the capabilities of existing training devices are constraining to 
the current training programs. 

The selection of a contractor, familiar with ISO procedures 
but without specific experience in the application of the model, 
was intended to serve two purposes: (1) an evaluation of the 
clarity and utility of the Data Item Descriptions (DIDs) and the 
specification, i.e. determining how accurately the model could be 
interpreted and implemented by the contractor; and (2) a source of 
recorranendations for modifications that will improve the procedures 
in future ISO programs. 

The analysis phase of the F-4 ISO program, which is discussed 
in this report, was applied in that the end products constituted 
the first steps in the process of developing an ISD-based training 
program and will be used as the basis upon which the design and 
development phases will be formulated. It was research in that it 
was the first application of a rigidly specified model by a con- 
tractor other than those who had developed the steps and procedures 
which were incorporated into the model . 

STATEMENT OF THE PROBLEM 

The primary goals of the analysis phase of the F-4 ISO program 
were to implement the ISO model in accordance with Specification 
N215-264 and to prepare and deliver the specified end products in 
accordance with applicable DIDs, Analyses were performed for both 
the Navy and Marine training courses and the F-4 J and N models. 
The starting point for this contract was the task of developing 
objectives hierarchies and Specific BeJiavioral Objectives (SBOs) 
from the Task Listing. This listing, which v^as developed by the 
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Marine ISD team at MCAS , Yuma under the guidance of Courseware, ^ 
Inc. was supplied to Allen Corporation at the beginning of the 
contract. 

The procedure specified by NAVTRAEQUIPCEN was to have Marine 
and Navy F-4 pilot and RIO subject iratter experts (SMEs) carry out 
those tasks in the ISD process which required knowledge of F-4 
operations, tactics, and procedures. This vas done under the 
technical guidance of Allen Corporation. Other tasks in which 
expert knowledge was not required were carri.^d out primarily by 
Allen Corporation personnel- The major tasks for whirch deliverable 
end products were specified were; 

1. Objectives Hierarchy and SBO .Development 

2. Media Selection 

3. Syllabus Development 

4. Training Support Requirements Analysis (TSRA) 
SYSTEM DESCRIPTION 

THE AIRCRAFT. The F-4 is a two-place, supersonic, long-range, all- 
weather fighter built by McDonnell Douglas Corporation. It is de- 
signed for intermediate and long range high altitude interceptions 
using missiles as the principal armaiTient and for ir<termediate or 
long range attack missions to deliver airborne weapons/stores . The 
aircraft is powered by either two single rotor, axial flow, variable 
stator turbojet J79-GE-8 or J79-GE-10 engines with afterburners. 
It Is capable of carrier operations. 

The degree of sophistication in modern weapons systems has 
necessitated the employment of two-man crews in many of the current 
fighters to minimize the probability of operator overload. Although 
there is a clear division of labor between the pilot and RIO duties, 
they must function as a:i efficient team to carry out the missions 
of the F-4. The training program for both crew positions must, 
therefore, stress individual duties, as well as, contain a strong 
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emphasis upon the teamwork required. The most successful fighter 
teams are those who work together continuously and know each 
other's reactions, strengths and weaknesses. 

THE MISSIONS. The F-4 has two primary missions, air-to-air 
interceptions and air-to-ground delivery of airborne weapons/stores. 
In addition, the F-4 performs armed esco'^t and reconnaissance missions - 
The primary weapons systems us^d to carry out these missions are the 
bombing equipment, the data link system, the direct radar scope 
camera, electronic countermeasure equipment, gunnery equipment, 
and the missile control equipment. Since the F-4 has been in the 
Navy/^!arine inventory f r a number of years, a large amount of 
experience in executing the above missions in combat, practice, and 
training environments has been accumulated. 

THE TRAINING PROGRAM. The existing Navy and Marine transition 
courses are the results of many iterations and refinements over 
the years since the F-4 was introduced. The Marine syllabus in- 
cludes training to 60 percent (combat capable) with the remainder of 
trainint, to 100 percent proficiency (fully combat qualified^ taking 
place in tactical group training at operational squadrons. Graduates 
of the Navy syllabus are 80 percent proficient with 20 percent left 
for tactical group training at: operational squadrons. A summary of 
the total training for Marine F-4 pilots and RIOs is srown in Table 1. 
Included is the training to 60 percent proficiency at MfAS, Yuma, 
plus the additional training provided at duty squadrons. Table 2 
provides a further breakdown of ground school training which currently 
takes place at VMFAT-101, MCAS , Yuma. Tables 3 and 4 are break- 
downs by sorties/hours and simulator periods/hours of the Combat 
Capable Training portions of the pilot and RIO syllabi. 

Trainees for the pilot transition course come from four 
Advanced Jet Training programs. RIO trainees come from the 
Advanced Naval Flight Officer school. Students for the refresher 
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TABLE 1. SUMMARY OF BASIC MARINE TRANSITION AND CONVERSION 
TRAINING FOR F-4 PILOTS and RIOs 



Weeks , Course Actlvi ty " 

1-3 Ground school ^no flight simulator Training group* 

training 

4-6 Air-to-air and air-to-ground MAWTU* 

courses 

7-26 Combat capable training Training group* 

27-38 Combat ready training Tactical group 

39-46 Combat qualification training Tactical group** 

47-54 Full combat qualification training Tactical group** 



* Takes place at FRS 
** Takes place at operational squadron 
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TABLE 2. 



BREAKDOWN OF TRANSITION GROUND SCHOOL AND 
SIMULATOR/FLIGHT TRAINING AT MCAS, YUMA 



Course 



Duration 



Ground School 



2 weeks 



Aircraft Systems 
Cross-Country Packets 
Aircraft Pref light 
Plane Captain Signals 
Safety 

Aerodynamics 
Finergency Procedures 
Course Rules 

Aircrew Teamwork and Responsibilities 
Basic Radar 
Navigation Computer 
Airborne Missile Control System 
Flight Planning 
Performance Data 
Fl ight/Simulator Training 1 week 
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TABLE 3. BREAKDOWN OF THE COMBAT CAPABLE TRAINING PORTION 
OF THE PILOr SYLLABUS 



STAGE 



FLIGHT 
SORTIES/HOURS 



SIM. 

PERIODS/HRS 



TOTAL 
TRAINING 



Basic Qualification 

Familiarization 4/4.8 

Instrument 4/6.0 

Formation 3/3.9 

FMLP 1/0.8 

All Weather Intercept 7/8.4 

Missile Fire 1/1.5 

Aerial Ref. 2/2.6 

Ground Attack 9/9.0 

Vis. Nav. 1/1.3 

BAr^- 6/5.0 

Fighter Intercept 2/2.4 

Fighter Weapons 7/5.6 

47/52.3 



Entry level skills 
8/12.0 



12/18.0 
1/ 1.5 



1/ 1.5 
2/3.0 

24/36.0 



12/16.8 
4/ 6.0 
3/3.9 
1/ 0.8 

19/26.4 
2/ 3.0 
2/ 2.6 
9/ 9.0 
1/ 1.3 
7/ 7.5 
V 5.4 
7/ 5.6 

71/88.3 
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TABLE 4. BREAKDOWN OF THE COMBAT CAPABLE TRAINING PORTION 
OF THE RIO SYLLABUS 



STAGE 



FLIGHT 
SORTIES/HOURS 



SIM. 

PERIODS/HRS 



TOTAL 
TRAINING 



Basic Qualification 

Familiarization ' 1/1.2 

Instrument 5/ 7.5 

Formation 1/ 1.3 

FMLP 1/0-8 
All Weather Intercept 10/12.0 

Missile Fire 1/ 1.5 

Aerial Ref. 2/ 2.5 

Ground Attack 9/ 9.0 

Vis. Nav. 1/ 1.3 

BAM 5/ 5.0 

Fighter Intercept 2/ 2.4 

Fighter Weapons 7/ 5.6 



Entry level skills 
8/12.0 



12/18.0 
1/ 1.5 



1/ 1.5 
2/ 4.0 



9/13.2 
5/ 7.5 
1/ 1.3 
1/ 0.8 
22/30.0 
2/ 3.0 
2/ 2.6 
9/ 9.0 
1/ 1.3 
6/ 6.5 
4/ 6.4 
7/ 5.6 



45/50.2 



23/31.0 



68/87.2 
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pilot and RIO courses fall into two categories, those who have 
not flown in an F-4 for 6-18 months and those who have not flown 
in an F-4 for more than 18 months. The training which these 
pilots and RIOs receive is a subset of the total transition 
course. Instructors Under Training (lUT) receive a special six- 
week JUT course in addition to refresher ground school and 
simulator training. 

During transition training at MCAS , Yuma, trainet^s are 
assigned collateral duties which require an average of two hours 
- per day. T'jese jobs include assignmentr in operations, adminis- 
tration, maintenance, intelligence, and logistics and embarkation- 
Other duties include operations duty officer, squadron duty officer, 
maintenance duty officer, and legal investigations. Due to these 
collateral duties trainees are not e/vpected to put in a "full 
training day". In the day-by-day schedule produced as part of the 
F-4 ISD program allowances are made for these collateral duties • 

Some trainees, but not all, may attend additional schools 
while undergoing FRS training. These schools are attended if the 
periodic requirements for completing the schools occur during FF(S 
training. Among these schools are Survival, Escape, Resistance 
and Evasion (SERE) School, Flight Instruments Refresher Course, 
and Water Survival Training. 

The combat capable training courses are designed to produce 
fully NATOPS qualified pilots and RIOs* Trainees are evaluated in 
oral and open and closed book exams, in normal and emergency 
procedures using the COT and WST, and in flight. 
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SECTION II 
INSTRUCTIONAL SYSTEM DEVELOPMENT 

BACKGROUND 

Traditionally, training programs within the military have been 
generated in-house by a number of instructor personnel tasked with 
the difficult problem of sorting through all available information 
and developing an approach to the classroom presentation of an 
assigned content area. The resulting instructor guides (outlines) 
and student handouts are, by necessity, based cn information within 
each individual's realm of experience. This experience is largely 
based on prior instruction and specialized operational missions. 
Of particular significance in such an approach to training is a 
failure to conceptualize the entire system in perspective. This 
results in a piecemeal, poorly-coordinated buildup of instructional 
units with retrofitting as the method of necessity for curriculum 
development. Furthermore, the Subject Matter Experts (SMEs) 
approaching the task usually do not have the benefit of expertise 
in the principles and application of instructional technology. 
The problem is complicated by the rapid turnover in instructor 
personnel. Often too, an cttempt is mode to piece together the 
in-house training materials, typical ly- i nstructor guides (outlines), 
with separately developed technical materials provided by a 
contractor to complement existing hardware. The classroom instructor 
is ultimately left with the task of integrating the curriculum 
while performing within the confines of the existing policy and 
procedure of the training command. 

As "a result of this piecemeal procedure, most instructional 
programs could be characterized as having one or more of the 
following problems: 

° Critical content is underemphasized or is left out. 

Instruction and testing are oriented toward general 
content knowledge rather than tov;ard specific know- 
ledge related to operational performance. 
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Learning outcomes and evaluaiion crii:er'^a are 
poorly defined. 

° Instructional methods and media are selected vTith 
little regard for relating subject matter content 
to media capabj 1 i t1 es . 

^ Subject matter content is i l^: trucior-speci f i c 
and changes as instructors change. 

^ There is little- if any, distinction oetween 
content and instructional design experts, i.e. 
content experts do everything. 

° Evaluation and update procedures are rarely 
f ormul ated . 

ISO evolved in response to these symptoms, as well as, the 
increasing demand for greater efficiencies in a time of tightening 
resources. It is an attempt to derive orderly procedures for 
the design, development, implementation, and evaluation of in- 
structional systemiS, and to provide operational guidelines for 
carrying out the procedures, 

Many models haye been formulated over the past 20 years, 
r^ost have contained analysis, design, development, implementation, 
and quality control phases tied together in a sequential flow. 
Some have been more prescriptive than others. Some have failed; 
others have been partially successful; but not one has received 
general acceptance. Although the progress has been slow, ISD 
technology is developing as data cn the various methods are 
c<^l lected . 

The model being implemented in the r-4 program, represents a 
large improvement over earlier approaches. It contains well- 
defined procedures for some tasks, while others are less defined. 
There are m.any areas for potential im»provement in the model. 
Identifying some of these areas i-s one of the goals of this program. 

ASSUMPTIONS 

The following assumptions underlie the ISD model which was 
used in this program. 
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The resultant training courses should teach knowledge 
and skills which are required on the job. 

° The courses when implemented should be effici^t in 
terms of resource requirements. 

° The instructional program should be based on clearly- ' 
defined objectives. 

° Selection of instructional methods "and media should 

be based on a mapping of method and media characteristics 
into the traim'ng requirements of the objectives. 

° Solid testing, evaluation, and revision procedures are 
essential to success. 

^ Subject matter content can be best supplied by 

personnel experienced in system operation. To insure 
optimal use of their expertise, training specialists 
should channel this knowledge of the system into the 
appropriate ISD formats. 

METHODS/PROCEDURES 

The "6.2 Model" is the state-of-the-art in ISD. It is the 
most precisely defined and procedurally accurate approach to ISD 
yet developed. The detailed specifications for the model follow 
a sequential flow beginning with task listing procedures and 
terminating with instructor training for implementation (Figure 1). 
The first step which :s not part of the model, is the problem 
analysis. The purpose of the problem analysis is to prov.ide 
background information on the strengths and weaknesses of the 
existing training system (if there is one). Using this information 
it can be determined if efforts should be made to upgrade the 
program and, if so, what the approach should be. Not all problem 
analyses identify ISD as the most appropriate and^ cost-effective 
option. -Other options are no action at all, updating the media 
used without doing "a full analysis, and performing an ISD analysis 
?/-)n selected portions of the program. In addition, the problem 
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analysis provides background information concerning training 
program users"and available tra^ining assets. This information 
assists the ISD contractor in evaluating the total environment 
in which the upgraded training program will be designed and 
implemented. . ' 

The first step in the ISD model rs to develop a task listing. 
This step helps insure that all tasks that should be included in 
the training program are identified. The task listing is used in 
a job analysis survey which is administered to personnel who are 
qualified in the system on which the ISD program is being executed. 
During the job analysis S:urvey., these personnel are questioned on 
the critical ity of each task to successful performance, the 
frequency with which the task is perfornjed on the job, the 
difficulty of the task, where the task was learned (i.e., in a 
school or on the job) and where the task should be taught. The 
-results of the job analysis survey are used to determine which 
tasks should be included in. the training program and the emphasis 
which should be placed on those tasks selected for training (i.e., 
indepth versus refresher training). 

Once the task listing has been validated and tasks have been 
selected for training, the tasks are used as the basis for 
developing objective hierarchies and Specific Behavioral Objectives 
(SBOs). This step involves analyzing tasks from the task listing 
in greater detail to identify all skills and knowledge which are - 
prerequisite to performing the tasks identified in the task listing. 
The. structure of the hierarchies is such that all the component 
objectives of a major task are arranged from the simplest and 
most fundamental knowledge up to the most complicated skill. Each 
entry in the hierarchies is an SBO which specifies decisions that 
must be made, pieces of information that must be learned, procedures 
that must be^executed, or skilled actitDns which must be performed. 
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For each SBO, conditions under which the objectives must be per- 
formed and the minimum acceptable performance standards are 
specified. By laying the objectives out in hierarchy diagrams, 
the subordinate, superordi nate, and coordinate ^relationships 
among the objectives can be clearly identified. The objectives 
hierarchies and SBOs provide the fundamental data base on which 
the training program will be structured. They also are the direct 
inputs into the next two steps in the ISD process, media selection 
and syllabus development. 

Media selection involves determining which of the candidate 
media is best suited for providing instruction for each of the 
SBOs identified in the previous ISD step. The output of the media 
selection process is a list of acceptable instructional media for 
each objective. Acceptable media. are listed in order of preference. 
The goal of this process is to select media which will enable the 
best possible trainee performance while minimizing costs. The 
procedures to be employed in media selection have yet to be 
standardized in the 6.2 model and are, thus, still evolving. In 
the F-4 program media were selected on the basis of the level 
of behavior expected of the student, the level of content being taught, 
the number of instances the student should see, the minimum dis- . 
play requirements, and the amount of memorization required. Other, 
more conventional , media selection models use procedures in which 
the display and fidelity requirements of each objective are mapped 
into the display and fidelity characteristics of candidate media- > 
The current version of the ISD model, MIL-T-29053, does not specify 
a specific procedure to be used. Rather it stipulates that an 
explicit media selection model shall be provided for approval by 
the contracting agency, 

Afrer lists of acceptable instructional media have been 
specified for the objectives, the objectives are grouped and 
sequenced to form the syllabus. Objectives are grouped into lessons 
and the lessons are sequenced to form units, which contain lessons 

^ 
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dealing with the same general subject matter content area. Units 
are sequenced to form the syllabus. The rerultant syllabus should 
possess three characteristics. First, objectives should be 
sequenced so^ that those objectives which are lower in a hierarchy 
are taught prior to higher level objectives in the same hierarchy, 
i.e., instruction in the prerequisite skills and kn^owledges precedes 
instruction in the more complex skills or decision processes. 
SeconG, whenever possible, the sequence should specify early hands- 
on learning; and practice and maintain an integration of academic 
instructicn and skills practice throughout the syllabus sequence. 
Third, in general, major tasks should be sequenced from easiest 
to hardest- When .feasible , however, those tasks which will reo'jire 
extensive practice' shoul d be introduced early in the syllabus to 
allow the required practice to be integrated throughout the remainder 
of the syllabus. Applyijng these three guidelines during lesson 
organization and sequencing yields a flow of instruction in which 
trainees first learn operating procedures, then practice the 
procedures, and finally perform the procedures in more complex 
situations in which decisions and integration with other procedures 
are required. 

At this point in the sequence of ISD steps that portion of 
the analysis phase which involves the production of end products 
which win be converted into instructional materials has been 
completed. The final step in analysis is preparation of a Training 
Support Requirements Analysis (TSRA). The TSRA provides an estimate 
of the personnel, equipment, material s , and services wKich will 
be required during subsequent phases in the ISD program. Separate 
estimates for design, development, implementation, and evaluation 
and maintenance are provided. The TSRA is used as a planning 
document for budgeting purposes. 

Due to the detail provided on the number of lessons per in- 
structional medium in the syllabus and the cost of production per 
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lesson in each medium, the TSRA can be used to identify relative media 
costs. By using relative costs, areas of possible cost savings can 
be determined. For example, the production of slide- tape materials 
is relatively costly compared to workbooks or mediated interactive 
lectures. Workbooks and mediated interactive lectures are in most 
cases alternative choices for slide- tapes. If there are costing 
constraints for production and revision of materials, some of those 
lessons for which tape-slide was first chosen may instead be 
presented in workbooks or mediated interactive lectures. 

Another area in which the TSRA provides information is 
faci 1 i ties requirements . This is particularly important in the 
planning for an implementation in which major modifications to 
' existing facilities or the construction of new facilities will be 
required. In order to have the facilities available when all 
materials have been developed and the course is ready for imple- 
mentation, preparation of facilities must begin as early as 
possible after the initial ISD a'^r'fyses have been performed. The 
TSRA is the jumping off point fr nitial analysis into the 
much more costly and lengthy design, development, implementation, 
..and revision phases. 

The design phase involves the development of lesson 
specifications for all lessons in the syllabus, A lesson 
specification contains the subject matter content and outlines 
the instructional strategy for each objective in the lesson. It 
is during this step that the information that will be presented 
in the course is assimilated and the strategies that will ba'used 
in the presentation of the lesson and testing of its content are 
determined. In addition to a statement of the subject matter 
content of each objective, a lesson specification contains helps 
that may be used in remembering or applying the objective, the 
number and types of examples required to support instruction of 
each objective, the practice and testing requirements, and 
graphic speci fi cations . 
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After the design phase is completed^ the development phase 
begins. During the development phase a number of tasks are 
carried on concurrently. ^The primary task involves converting 
the lesson specifications into lesson and testing materials. 
Authors use the lesson specifications in conjunction with lesson 
guides which explain how to produce instructional material for 
each medium being used. Lesson materials are produced in draft 
form and reviewed by the instructional psychologist. Prototype 
materials are then produced for small scale tryout. Based' on data 
collected during the small scale tryout, lesson materials are 
revised for final production . 

While the instructional materials and tests are being 
developed, quality control and implementation plans must be form- 
ulat'"' These plans help insure that implementation, evaluation, 
and revision procedures are wel 1 -formul ated and are workable within 
the training environment. The quality control plan presents the 
procedures to be followed in continuously assessing the relevance, 
effectiveness, and currency of the instructional materials and 
the instructional management system. It provides for such 
assessment from the earliest materials development stage to the 
implementation of the training program and throughout the life 
cycle of the weapons system. The quality control plan addresses 
three major areas: 'Normative revision, internal quality control 
(trying out the finished materials with real students in the real 
training environment), and external quality control (the validity 
of the training program in terms of its relevance to operational 
requirements for skilled personnel). 

The implementation plan details the procedures and milestones 
which will be followed during the implementation phase of the 
ISO program. The plan focuses on training management and 
administration, personnel requirements, and implementation procedures 
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Some of the specific content areas^ contained in the implementation 
plan are: 

^ Instructional system integration 

^ Equipment storage, maintenance, and utilization 

° Student grading evaluation 

° Training materials management 

^ Faci 1 i ties util ization 

° Student scheduling and management 

° Schedule contingency plan 

The final step during the development phase is the preparation 
of an Instructor Training Course (ITC). The ITC is designed to 
teach Fleet Readiness Squadron (FRS) personnel to properly use 
the instructional system* It provides instruction in the following 
areas : 

^ The training organization to include personnel, 

organization, policies* and student characteristics 

^ The training program to include syllabus organization 
and student flew, instructor duties, media used in 
the t^^aining program^ student learning center operations, 
ISO team duties, and an overview of the ISD process. 

° Training system operation and scheduling 

^ Storage, operation, and troubleshooting of media 
devices, 

^ Conducting lessons in all media used in the program 
^ Evaluation of students and instructional materials 
^ Revision of instructional materials 

After all instructional materials have been produced, the 
quality control and implementation plans have been formulated, 
and the ITC has been prepared and administered, the implementation 
phase begins. The length of implementation is a function' of 
course duration and the interval between class convenings. At a 
minimum it is the length of one class, during which external 
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quality control data is collected on that class. Preferrably 
its duration exceeds that of one class so that da^a on mi'ltiple 
classes can be collected and necessary modifications to the 
training materials and/or the management plan can be made. Most 
of the "bugs" should be worked out of the new training system 
during this initial implementation phase. 

The final phase in the ISD process is evaluation and main- 
tenance of the training program for as long as it is in use. 
While this phase is less formal than the previous phases in the 
concentration on ISD methodology, it embodies a continuation of 
ISD procedures in the constantly ongoing process of updating the 
training program. This is a particularly crucial phase since '^'ne 
structure of the program will change and possibly erode if c nigh 
quality maintenance program is not provided. The procedui^es de- 
tailed in the quality control and implementation plans and the 
instructor training course must be maintained. This function may 
be allocated to a small ISD team which should bp well-schooled 
in ISD procedures in general and in the specifics of the 
particular training program. 

The scope of this F-4 ISD program encompasses the analysis 
phase of the ISD process. The task listing was generated by 
MCCTRG-10 ISD personnel under the guidance of Courseware, I 
It provided the primary input into hierarchy and SBO develop:.»^.nt , 
which was the first ti\sk of this contract. Subsequent tasks were 
media selection, syllabus development, and training support 
requirements analysis. How these tasks were carried out is 
discussed in Section III, Implementation, 



EKLC 



29/30 

3^ 



NAVRAEQUIPCEN 77-C-0081-1 



SECTION III 
IMPLEMENTATION 



ORGANIZATION AND STAFFING 

PERSONNEL. Three organizations participated in the F-4 ISO 
program. These organizations and their roles are as follows 



0 
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Contractor -On-site training, guidelines, 

and review of products in ^ 
accordance with ISO procedures 

MCCTRG-10 ISO Dept. -Supply subject matter experts 
(supplemented by to- provide input on F-4 procedures, 

VF-121) tactics, etc. Provide facilities 

for the project, 

NAVTRAEQUIPCEN -Project coordination, direction 

and monitoring 



Under provisions of the contract MCCTRG-10 was tasked to 
provicie Marine subject matter experts (SMEs) to the program. Four 
SMEs participated on a part-time basis for the first 26 weeks of 
the 32-week program. SMEs were not required for the final six 
weeks. In addition, three Navy SMEs from VF-121, NAS Miramar, 
participated cn a non-overlapping basis for a total of ten weeks. 
On-site contractor personnel included an instructional psychologist, 
who was the on-site supervisor, a former F-4 RIO, who assisted 
the on-site supervisor in coordinating all project activities and 
who reviewed outputs from the Marine and Navy SMEs, and a secretary. 
The program director was located off-site. He made monthly visits 
to MCAS, Yuma, reviewed all products of the program, and was 
responsible for liaison with NAVTRAEQUIPCEN. Overall responsibility 
for the program rested with the Allen Corporation Technical 
Director. He reviewed all products of the program. The hierarchical 
ErJCc^ organization of the program team is shown in Figure 2. 
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As indicated by the dashed lints in Figure 2, the activities 
of the Marine and Navy SMEs were governed by the existing military 
command structure. They reported to the head of the MCCTRG-10 
ISD Department, who was responsible for coordination with on-site 
contractor personnel . 

PROJECT SEQUENCING. The sequencing of major project activities 
was planned at the outset as shown in Figure 3. The project was 
contractually scheduled to be a 32-week effort. Hierarchy and 
SBO development was scheduled for U weeks. Media selection, 
syllabus development, and the TSRA were scheduled for four weeks 
each. The right sides of the hierarchy and SBO development, 
media selection, syllabus development, and TSRA blocks indicate 
the submission date for a draft report. In each case the final 
report for that activity was scheduled four weeks after draft 
report delivery. 

PROJECT ACTIVITIES 

INTRODUCTION. In the following sections detailed descriptions of 
each of the major ISD activities conducted during the project 
will be discussed. For each activity t' - following categories 
of information are included: 

° An overview of the purpose of the activity 
^ A description of the procedures employed 
° A description of SME training 
^ An account of the problems and solutions 
The last section contains a summary of the F-4 training 
courses developed during this program. 

OBJECTIVES HIERARCHIES AND SBOs. 

Overview , The first major task performed under this contract 
was the development of objectives hierarchies and SBOs. A 
o hierarchy is a visual layout of the structure of the component 
iS^^*^ objectives which support the performance of a task. It shows 
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subordinate, superordinate , and coordinate relationships among 
the objectives. The procedures used to develop hierarchies of 
objectives and tasks provide a systematic way of determining 
what the student must learn in order to perform his job. They 
constitute the data base which will be use?d in the further 
development of the training program, A training program developed 
from objectives hierarchies is more likely to be complete in its 
job and task orientation and free of irrelevant information, which 
is time consuming tc teach but adds little to the student's 
ability to perform the job- 

Procedures . The input to the hierarchy and SBO development process 
is the validated task listing. It contains the major tasks whiCh 
have been selected for training during job analysis. For this 
program the pilot and RIO task listings were furnished by the 
government. They were structured by missions and segments within 
missions. There were four missions with ten mission segments per 
mission. A visual representation of. the mission by segment 
structure of the task listing is shown in Figure 4. The entries 
in the cells are the mission by mission segment numbers. These 
numbers formed the basis for the numbering system which was used 
during hierarchy development. 

The procedures used to develop the objective hierarchies are 
outlined in Figure 5. An elaboration of the steps shown in Figure 
5 follows: 

Step 1: List major tasks - This step was accomplished during 
task listing. It is included here to emphasize that the task is 
the input to the objective hierarchy and SBO development process. 

Step 2: Select the most, or next most complicated tas k 
for analysis - The following criteria should be used in the selection 
of the most complicated task: 

^ Select a task which involves a large part of the job 
time 

^ Select a task which requires many sub-procedures for 
its completion 

'^7 



SEGMENT 



" 1 

II .1 

11 Pre- 
Mission "fission 
No. Name Planning 


.2 

Brief 


.3 

Pre- 
Launch 


A 

Takeoff I 
Departure 


.5 

Nav. 


.6 

Tactics 


.7 

Refuel 


.8 

Approach 
& Landing 


.S 

Post 
Mission 


.10 
Debrief 


1! 

LO Air to ; II 

Surface ; 

II 
II 


1.2 


1.3 


1.4 

1 , 


1.5 


1.6 


1.7 


■ 

1.8 


— 

1.9 


1.10 

1 


r—^ 1 

2.0 Air to 0 1 
Air ; 

11 

II 


2.2 


2.3 

■ 


2.4 


2.5 


2.6 


2.7 


2.8 


2.9 


2.10 


n — ■ ■ 

3.0 Recon^ | ^ | 

naissance ; 

11 
II 


3.2 


3.3 


■ 

3.4 


3,' 


3.6 




3.7" 


3.8 


3.9 


3.10 

1 

1 
1 

■i 


n — ■ 

II 

4.0 Escort 4,1 

11 


4.2 


— 1 

4.3 


4.4 


4.5 


4.6 = 


4.7 


4.8 


4.9 


4.10 

1 



I 

Figure 4. Structure of the F-4 Pilot and RIO 
Task Listings 



NAVTRAEQUIPCEN 7/-C-0081-1 







9 














[ vcs 























Figure 5, The Process of Objectives 
H i era rchy De vel opment 



4 3 



37 



NAVTRAEQUIPCEN 77-C-0081-1 



^ Select a 'task which tends to be one of the most 
difficult 

Step 3: List all components of the major task - Analyze 
the performance components of the selected major task and^jdivide 
the major task into its -component subtasks. This will yield the 
steps or procedures required to perform the major task. Steps 
or procedures may be decisions, as' well as, overt, observable acts. 

Step 4: Determine whether all necessary decisions have been . 
identified as task components - When analyzing a task that requires 
performance of a large number of procedures and interaction with 
complex system hardware, identification of each operator decision 
is made to determine whether the operator must decide when to 
perform a procedure, decide which of multiple rules or procedures 
to use, evaluate the adequacy of a procedure, and/or decide when 
a procedure should be stopped. In analyzing a major task, decision 
subtasks can easily be omitted. This step is included as a cross- 
check of Step 3 to insure that all decision tasks have been 
included. 

Step 5: Add decision components to the list - If additional 
decision subtasks are identified in Step- 4, they should be added 
to the component subtask list for the major task. When incor- 
porated into the hierarchy structure, these decision subtasks will 
be on the sane level as the performance subtasks. 

Step 6: Determine whether there is a significant memorization 
r equi rement - the memorization component is significant if an 
average trainee would be unable to perform the task as a whole 
because he could not remeniber which components must be performed 
or the order of their performance. If the memorization component 
for the task is significant^ it will be incorporated in Step 7. 

Step 7: Add a component requiring recall of the steps of 
the nftajor task - If there is a significant memorization component 
identified in Step 6, it should be added to the component subtask 
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list for the major task. When incorporated in the hierarchy 
structure, the memorization subtask will be at the same level 
or one level below the performance subtasks. 

Step 8: Determine if there are too many components - The 
component subtask list is examined to determine if any subtask 
repeats another subtask on the list, is not necessary to the 
accomplishment of the main task, or is trivial- The component 
listing should prO^vide the level of detail which will be required 
when the objectives written from the listing are used as input 
for the production of segment specifications in the design phase, 
i.e., the objective name^sjiould clearly indicate to a subject 
matter expert what informatron will be contained in the segment 
specification for that objective. If this information is in- 
dicated by a higher level objective, an objective below in the 
hierarchy would be trivial. On the^.other hand ff the objectives 
in the hierarchy do not identify comp^^ely the information that 
should be contained in segment specif ica\ions , additional 
objectives must be added. The addition o/Nsubtask components is 
discussed in Step 11. \^ 

Step 9: Narrow the set of components - Thev^inal listing 
should contain the minimum set of subtasks requireM^to perform 
the task, consistent with the requirements for segmerv^ specific 
cations discussed above. To accomplish this, overlapping and 
non-essential subtasks should be deleted and where appropriate 
multiple trivial subtasks should be grouped into a single relevant 
subtask. 

Step 10: Determine if there are too few components - If 
after having mastered all given subtasks, the trainee would be 
unable to perform the main task without more than a few simple 
instructions, one or more subtasks have been omitted. A subtask 
left out at this point may result in instruction left out later, 
i.e», lesson and segment specifications will not contain the 
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totality of information required to perform' the task. 

Step 11: Expand the set of components - tf.^the component 
subtask listing contains fewer than the minimum number of sub- 
tasks required to perform the task, it should be expanded to 
insure that the completed training program will contain al"l^ir^- 
formation necessary to train students to perform the task. 

Step 12: Write an instructional objective for each componen-y - 
An instructional objective contains three elements, the subtask 
component statement, which is the "action*' portion of the objective, 
the conditions, and the standards. Conditions describe the 
circumstances under which the subtask will be performed. Types of 
conditions statements are system configuration, environmental 
factors, equipment factors, relevani: available information, and 
initiation stimulus. Standards are the criteria for minimum 
acceptab'ie subtask performance. Types of standards statements 
are time, accuracy/error rate, degree of conformity with specified 
procedures, and safety considerations. 

Step 13: Arrange objectives in the hierarchy - Objectives 
are arranged under the superordi nate objective that is being 
analyzed. All are one level below the superordinate objective. 

Step 14: Determine whether each objective is a basic level 
behavior - An objective is a basic level behavior if it is the 
lowest level behavior that is significant for training. If all 
objectives are basic level behaviors, the next complicated major 
task is selected from the task listing and the process is repeated. 

Step 15: Select one basic level subtask - If one or more 
objectives are not basic level behaviors, one objective is chosen 
and subtask components for that objective are listed. This 
process is iterated until all of the lowest level objectives have 
been identified for the major task. 

Examples of a hierarchy diagram and an SBC page are shown in 
Figures 6 and 7 respectively. A detailed description of the 
numbering and' coding conventions used in the hierarchies and 
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formulated. 

The determination of conditions and standards during 
hierarchy and SBO development detracts from the primary purpose 
of structuring hierarchies which provide the framework for the 
development of the syllabus and the ultimate training program. 
It is recommended that consideration be given to deleting conditions 
and standards from SBOs written in conjunction with hierarchy 
development and emphasizing the importance of including complete, 
accurate conditions and standards in the segment specifications, 

MEDIA SELECTION. 

Overview. The next step in the F-4 ISO program was to select 
appropriate instructional media for each SBO produced during the 
preceding hierarchy and SBO development step. This step provides 
a matching of the instructional capabilities of various media with 
the characteristics of the objective as they relate to presentation 
and learning strategies. It is important that this mapping of 
objective requirements onto media capapbi 1 i ties be as accurate as 
possible within cost and feasibility constraints. This helps 
insure that trainees are presented information, afforded practice, ' 
etc., in an optimal learning environment. 

The algorithm used to select media differed from more 
conventional media selection procedures in that level of expected 
behavior, level of content, amount of practice required, and the 
amount of memorization required were explicit criteria for 
selection. In addition, presentation display requirements, the 
nucleus of the more conventional media selection models, was 
included as one of the components of the algorithm. 

Procedures . The input to the media selection process was the 
listing of all SBOs. Each SBO was carried through the selection 
algorithm to determine the primary and alternate media which 
should be used. Alternate media were provided for each objective 
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to allow flexibility during later phases of the ISD process when 
modifications based on cost factors or syllabus organization may 
be required. 

The media selection model used in this program involved 
asking a series of five questioriS, the answers to which traced a 
path through a network to a terminal block which contained a 
code number for the media selected. The network is shown in 
figure B and the questions used to determine paths through the 
network arc shown in \'ii]urc 9. The question blocks in the network 
are labelled Ql, (JZ , QJ, ^4, and g5. 

The selection process begins at the triangle in the center 
of the network and goes to question one (Ql)- The numbered 
response to Ql determines the path that will be followed to the 
next question. In the same manner:, at each of the other four 
questions (Q2 through Q5) the procedure continues along the path 
associated with the appropriate answer to each question. 

Each path leads to recommended media, which a>^e the terminal 
blocks Ml through M44 on the network. The media selections 
corresponding to the numbers in the terminai blocks are presented 
and defined in Appendix B. In addition. Appendix B contains 
examples of completed media selection diagrams. 

The media selection process was conducted primarily by the 
project instructional psychologist. SMEs , however, provided 
significant input particularly relating to the difficulty of 
the objectives- Questions 2 (amount of presentation or practice) 
and 5 (memorization requirements), in essence, assess the 
difficulty of the objective. SMEs were also called upon to 
answer questions relevant to objective content. 

SME Training . Since media selection was primarily a contractor 
task, no extensive SME training was required. An informal four- 
hour course was conducted to acquaint the SMEs with the selection 
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QUESTION 1: 

What is the level of behavior expected of the student 
in this segment? 

1 = familiarization 

2 = discriminated recall 

3 = rul e-using 

QUESTION 2: 

What level of content is being taught in this segment? 

1 = f ami 1 iari zati on 

2 = paired associate 

3 - concept 

4 = rule 

QUESTION 3: 

Is the miniinurn critical set of ins-^^nces the student 
needs to see smal 1 or large? 

1 = small 

2 = large ^ 
QUESTION 4: 

What is the minimum display requirement? 

1 = verbal and/or symbolic and/or static simple 

pictorial 

2 = verbal and/or symbolic and/or static complex 

pi ctori al 

3 = dynannc pictorial 

4 = interactive 

QUESTION 5: 

Is the memorization component of this objective large 
or small? 

1 = small 

2 = large 



Figure 9. Questions and Answers Used 
in Conjunction with the 
Network Shown in Figure 8 
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process and to relate media selection to other steps in the 
ISD process. It was necessary that the SMEs understand the 
procedures in the model so that they could answer questions 
v^hich required subject matter .experti se. 

Problems and Sol ut ions . The primary probl em in apply i ng the 
media selection model was the ambiguity of some of the responses 
to the five questions. Without more precise, working definitions 
than those supplied in the F-4 ISD specification the model could 
not be applied with consistency or with the accuracy requ^'^ij^'. 
In order to remedy this problem, Allen Corporation "ST ersor 
consulted with the Courseware, Inc. personnel who fcrr::^iated 
the selection model. The resultant definitions of the ~enrs 
made the model workable; however, it was still felt that further 
clarification was needed- The five questions and elaborations of 
the possible responses to each are as follows; 
Question 1 : What is the level of behavior expected of the 
student in this .segment? 

1. familiarization: the student will be exposed to the 
information or skill, but need not be intimately 
familiar with it and will not be tested. 

2. discriminated recall: the student will remember, recall, 
or recognize something based on previously presented 
information, e.g., recall operational fuel flow rate. 

3. rule using: the student will apply a rule to a given 
set of inputs to formulate an output, e.g., dis- 
criminate among types of threat aircraft. 

Question 2 : What level of content is being taught in this segment? 

1. familiarization: the content level is for basic 
acquaintance and recognition but will not be a 
test item." 

2. paired associate: this content level involves the 
teach i ng of a one-to-one association ( presence of 
A indicates 8 is occurring) and would require 
future recall of the content. 
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3. concept: the content level would show grouped or sets 

of symbols or events with common critical characteristics 
or names, e.g.^ identify an engine fire. 

4. rule: this level of content requires the manipulation 
of a concept, e.g., calculate refueling requirements. 

Question 3: Is the minimum criticcil set of instances the^ student 

needs to see small or large? 

Essentially the question was inte^^preted to say, "How often 
do we have to expose the student to this content during his 
traini:ig cycle in order for him to master the content cr^ 
how often must a given skill be practiced to insure mastery?'' 

(Question 4: What is the minimum display requirement? That is, 

how sophisticated must the presentation to the trainee be in ordet* 

to adequately convey the desired content? 

1. verbal and/or symbolic and/or static simple pictorial, 
e.g., a simple photograph, typed or recorded statement," 
or drawing. 

2- verbal and/or symbolic and/or static complex pictorial, 
e.g., a complex photograph, typed or recorded statement 
or drawing. 

3- dynamic pictorial, e.g., a moving picture, animated 
drawing, computer generated image, etc. 

4. interactive: Must the presentation media sense student 
responses and respond appropriately? 

Question 5 : Is the memorization component of this objective large 

or small? 

This c^uestio^l was interpreted as a discrimination between 
"small" and "large" based u[)on content complexity and volume. 
The more ccjmplox objectives involve certain detailed 
memorization factors, e.g., navigation would contain 
components re^juiring "large" memorization. Other objectives 
that are assisted by supportive documentation such as 
checklists, were categorized as containing a "small" 
memo.ri zation component. 
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If this media selection model is to be used in future ISD 
programs^ It is recommended that n^ore precise operational definitions 
be used in place of those currently supplied. Incorrect inter- 
pretations of the five questions and their sets of responses can 
have significant effects on the media selected. This could lead 
to teaching objectives in media which are not optimally suited to 
the learning requirements of the objectives. 

The possible set of media which could be selected for use 
in the F-4 training program was constrainted to the candidate media 
contained in the model. This constraint led to a "closed" selection 
process rather than an ''open*' process in which a much larger set of 
candidate media could have been considered- For example^ among 
other feasible alternatives for this program are audio tapes with 
workbooks and continuous loop 8 mm. systems, which can provide 
either frame-by-frame projection or continuous motion. 

The area of media selection is a difficult one at best. 
There are almost as many selection models as there are media types. 
As yet, there is no completely satisfactory approach to this 
problem. This contractor has not developed en alternative ~ broach " 
to media selection; however, within the context of the F-4 . jel 
several suggestions can be made. Rather than selecting specific 
media it is recommended that a class of media be selected, all 
members of which have similar presentation capabilities. Once 
a class of media is selected for an objective, trade-off studies 
can be performed to determine which medium in the clas'^ is best 
suited in terms of specific capabilities and cost factors. Cost 
is not considered in the present model and should be evaluated 
during the media selection process. For example, data in the 
Training Support Requirements Analysis indicate , that the relative 
cost of slide tapes and workbooks are about four to one. Yet 
slide-tapes and workbooks are similar in instructional capabilities. 
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In the F^4 ISD specification, media selectft)nr and syllabus 
development are discrete, sequential processes wit-h media selection 
preceding syllabus development. They are, thus, two independent 
processes w"th the results of media sel ection • having minimum impact 
on lesson organization and sequencing. The structuring of lessons 
is done on the basis /of subject matter content rather than on the 
basis of media selected for individual objectives. Under the 
current specification lessons are organized by content. The media 
for the objectives /in the lesson are then examined to determine 
if more than one medium was selected for the grouped objectives. 
If so, the predominant medium from those assigned to the objectives 
in the lesson is selected. This pro«_edure is inefficient since 
some of the media iselected for ind? vidua! objectives are changed 
.to be consistent with the lesson content requirements. Media 
selected for objectives, grouped into a lesson on the basis of 
content and context of training, should fall within the same 
class. If this does not happen,, it is indicative that the lesson 
has been poorly organized- 
It is recommended that media be selected for lessons rather 
than for individual objectives,' thus reversing the order of media 
selection and syllabus development. Two advantages of this approach 
are: (1) the process has to carried out fewer times since there 
are fewer lessons than there are objectives and (2) once made, 
media selections will be final ra-ther than subject to change 
during a subsequent task in the iSD process. 

SYLLABUS. / 

/ 

/ 

Overview .^ /The development of pilot and RIO syllabi was the last 
step in the -analysis phase v/hich provided input directly into 
lesson specification development (Figure 1). The other remaining 
analysis step was the Training Support Requirements Analysis 
(TSRA) which provided personnel, materials, and services cost 
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estimates for the four remaining phases* but which was not directly 
related to the subsequent^ phases in terms of technical steps 
specified for course development. 

Syllabus development was the process of grouping the SBOs 
from the hierarchy and SBO development step into lessons, and 
sequencing the lessons to form the pilot and RIO syllabi. The 
resultant syllabi formed the basis for continued development of 
the two training courses. Each lesson constituted a discrete 
block of instruction with the objectives contained in the lesson 
providing an outline of the subject matter content to be covered 
in ground school or the skills to be practiced in simulator and 
flight lessons. The goal of syllabus development was to arrange 
all objectives so as to facilitate administration, scheduling, 
sequencing, testing, and learning. 

Procedures * Each course was structured into three levels of 
organization. These levels ordered from smallest to largest are 
as follows: 

^ Segment - The instruction required to teach one objective. 

A segment is comparable to an objective and is 
worded the same as an objective. During syllabus 
development the terminology is changed from 
"objective" to "segment" to indicate that each 
objective is now thought of as a segment of 
instruction. During lesson specification 
each segment will be the starting point for 
determining the subject matter content and 
instructional strategy which will be used to 
teach the objective. 

^ Lesson - A group of related segments with a required 
testing point. The F-4 ISD specification 
contains guidelines for the number of segments 
which should constitute a lesson. The range 
is from 3 to 15 objectives. More imporuantly, 
a lesson should be composed of segments v^-hich 
form a logical grouping in terms of content 
and can be taught at* one sitting. 
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Unit - A group of- related lessons with a required 
testing point. This is the largest sub- 
*d1vision within the syllabus- Units contain 
lessons grouped on the basis of subject matter 
content and ordered so that earlier lessons are 
prerequisite to latter lessens ijti the same unit. 
The structure of units thus has a large impact 
on the effectiveness of training. Students 
should have mastered prerequisite skills and 
knowledge before being exposed to more compli- 
cated learning situations . For example, 
maneuvers to be performed in the aircraft 
should first be discussed in a ground school 
le? - i atic .r/^rticed in a simulator (if simulator 
capabilities ^it) before befng practiced in 
Che e^'^craft B> -using this sequence the trainee 

be - pr^fjarec to make efficient use of the 
iiK, t: x- 3ns1ve training dwice, the aircraft- 
Finci ly, in sequencing units within the syllabus, 
the lame ci^icelines as those u-sed in sequencing 
lessc"' >r*tnip jnits shOL^ld be used, -i.e,, establish 
prerequisites oy ur":ts- For example, a unit on 
b^::ic aircraft maneuvering should precede a unit 
on ^ i r- to -air combat. 

The total . syl labus vel oprr.ent process has two components, 
organization of segments into lessons and sequencing lessons to form 
the syllabus. Steps used in organization and sequencing proceeded 
from flight lessons, to simulator lessons, and finally to ground 
school lessons. The flight syllabus was formulated first and pro- 
vided the frantework around which the remainder of the syllabus was 
organized and sequenced. The specification for the F^4 ISD program 
cpntained guidelines to be used in developing t'^e flight syllabus. 
These guidielines were as follows: 

1. Determine from the problem analysis data the maximum 
feasible number of flights that can be contained in the 
syl la bus . 

2. Identify any content requirements, ordering requirements, 

or other constraints which are imposed by readiness training 
squadrons or higher authority. 

3. Extract from the objectives hierarchy all objectives 
designated during media selection as requiring aircraft 
flight- 
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4. Examine the objectives to identify common logical 
groupings * 

\ 

"5. Sort the pilot and RIO flight objectives into aircraft 
\ flights within imposed constraints. Further guidelines 
Vfor sorting the objectives are listed below: 

(a) Structure the first flight so that it contains 
the absolute minimum number of objectives required 

\^to have a flight. 

(b) Iti subsequent flights add new tasks so that the 
fVlghts proceed from easier to more difficult. 

(c) For each flight, group objectives so that there 
are sufficient objectives to warrant a flight 
but not \SO many that each cannot be properly 
eval uatecl^ 

(d) Include neW\ ob jecti ves in each flight. 

(e) Structure flights so that when a new objective 
is introduced , \i t is practiced and evaluated 
in that flight, in several other immediately 
following flights, and then again at periodic 
intervals as needed^ to maintain proficiency. 

(f ) Group objectives intb^ fl ights so that the 
prerequisite cl assrooriK and trainer periods for 
each flight form a set Vf instructional blocks 
which are reasonably unified in subject matter 
and approximately equal ih\ si ze. 

The guidelines outlined in 1-5 above were closely followed in 
developing the flight syllabus. Based on problem analysis data, 
supplemented by current figures on aircraft availability, it was 
determined that the number of flights in the ISD syllabus should 
be in accord with the number of flights in the current syllabus. 
Several squadron constraints were incorporated into the syllabus. 
These included the following: 

1. Completion of the familiarization and flight support 
ground school and trai ner i nstructi on as speci f i ed 
in the Training and Readiness Manual , plus open and 
closed book exams, prior to the first flight. 

2. Completion of carrier qualification ground school 
and 50 flight hours prior to carrier qualification 
flights. 

3. Ten flight hour*^ in the aircraft prior to the f1r?;t 
ni ght f 1 ight . 
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4. Fifteen hours in the aircraft prior to firing the 
first air-to-air missile. 

5. Fifteen hours to include one night flight in the 
aircraft prior to the first cross-country. 

Objectives selected for flight lessons during media selection 
were extracted from the media selection report. These objectives 
were examined for common logical groupings, based primarily upon 
the hierarchy structure. 

Due to training requirements and simulator constraints, 
strictly adhering to sorting guidelines (a)-C^') above was found 
to be infeasible in many cases . For example, it was judged to be 
training efficient to include most flight objectives for the air- 
to-air phase in the first air-to-air flight- This was done because 
the employment of different tactics and maneuvers is a function 
of relative aircraft positions and in practice primarily involves 
decision processes based on the situation. Subsequent flights, 
■*"^^^n, involved the selection of the appropriate maneuvers and 
u ,,tics from those introduced 'n ground school and in the first 
air-to-air flight. Since mo5.t objectives were specified in the 
first flight, the air-to-air portion of the syllabus was not 
integrated so that each flight was proceeded by ground school and 
simulator lessons. Rather ground school lessons were grouped at 
the beginning of the air-to-air phase followed by a m.ixing of 
simulator and flight lessons. 

Simulator constraints had a significant effect on formulating 
the air-to-ground portion of the syllabus. Since the Weapons 
System Trainer (WST) has no visual system, all air-to-ground 
practice was specified for flights. As with air-to-air training, 
air-to-ground, ground school was placed before the flights. The 
first air-to-ground flight contains most of the air-to-ground 
objectives, with subsequent flights requiring decision processes 
based on the tactical situation. The grouping of initial ground 
school, fo^owed by practice, is sound in terms of learning theory. 
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i.e., provide the student with the information required to select 
among alternatives, then place him in problem-solving situations 
which require decisions based upon the information. 

Despite the dif -^i cul ties encountered in following the sorting 
guidelines for certain portions of the syllabus, both the pilot 
and RIO syllabi were generally structured in accordance with the 
guidelines. Starting with the first flight which contains a 
minimum number of basic objectives the flow proceeds from easier 
to more difficult. Flights contain enough objectives to warrant 
a flight, but not so many as to preclude effective performance 
evaluation. Practice of objectives is spaced so that after initial 
introduction objectives are practiced in immediately succeeding 
flights and periodical thereafter. Consistency in subject matter 
content is maintained from prerequisite ground school, through 
simulator lessons, to flights. This is the most fundamental 
principle in the sequencing of instruction. The student should 
first thoroughly know and understand the procedures required for 
the performance of a task. This initial academic instruction 
should be followed when feasible, by ground-based practice to 
develop task skills. The task skills should ther be practiced 
and refined in the aircraft. 

Following the development of ^* e flight syllabus the 

simulator exercises and ground school lessons were organized and 

sequenced around the fran^ework provided by the flight syllabus. 

Guidelines provided in the specification for simiuator exercises 

were as f ol 1 ows : 

1- Precede each aircraft flight with a crew coordination 
trainer exercise during which all Oujecti\'es to be 
practiced during the flight are practiced and evaluated. 

2. Identify new crew coordination simulator'" exercises 
for those objectives which can be performed in a 
trainer but not in flight, e.g.^ emergency procedures. 
After initial practice intersperse these objectives 
throughout the remaining trainer exercises. 
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3- Sequence .emergency procedures trainers prior to 
the first flight. 

4. Develop individual position trainer exercises to 

support each crew simulator exercise- 
Guidelines for organizing and sequencing ground school lessons 
were as follows: 

1. Use the objectives hierarchies to identify objectives 
that are prerequisite to the first trainer exercise. 

2. Group objectives into a series of lessons following 
criteria for good lesson structure, i.e., subject 
matter content, sequencing within lessons, and length. 

3. Use the objectives hierarchies to identify objectives 
which are prerequisites the other simulator and 

ai rcraf t exerci ses and group the objecti ves i nto 
series of lessons. 

Although the organization and sequencing of simulator lessons 

and ground school lessons were specified as sequential steps, it 

was found that executing the two steps in parallel facilitated 

overall syllabus development. This was true because in many cases, 

for training efficiency, it was deemed desirable for simulator 

lessons to be interspersed between ground school lessons rather 

than between ground school and flight lessens. In other cases 

ground school lessons directly preceded flight lessons or flight 

lessons were grouped without intervening ground school or simulator 

lessons. Deviations from the guidelines for organizing and sequencing 

simulator and ground school lessons did not result in a departure 

from the fundamental philosophy of syllabus organization stated 

above, i.e., instruction in a task should proceed from academic 

instruction to skills practice in a s:mulator to skills practice 

In the aircraft. 

The final ste[) in syllabus development was to divide the 

sequence of lessons into units. By definition a unit which 

contains a flight should end with that flight, thus the next 

lesson is the first lesson of the next unit. In the pilot and 
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RIO syllabi some units contain multiple flights. When no ground 

school or simulator lessons were scheduled between multiple flights 

and the same or similar skills were practiced in the successive flights, 

no divisions between units were specified. Unit boundaries should 

be based more on subject matter content distinctions rather than 

upon arbitrary divisions based on instructional media. 

SME Training , Extensive SME training was not required for syllabus 
development. A one-day seminar was conducted to acquaint the SMEs 
with the guidelines to be used and to formulate specific procedures 
to be used in developing the pilot and RIO syllabi. 

Problems and Solutions . Syllabus development was the most difficult 
task carried out in this program. The fundamental problem was the 
lack of agreement among SMEs on the optimal organization and 
sequencing of objectives. Each SME had a different perspective on 
how the pilot and RIO syllabi should be structured. This led to 
lengthy discussions which produced little progress. In order t6 
facilitate the development c^^ocess, the recently revised VMFAT-lOi 
Training and Readiness (T z^i R) Manual was used to provide 
guidance. The syllabus outlined in the T and R manual basically 
follows the rules of good sy^.abus structure. By using it as a 
guide, consensus among the SMEs was achieved, and syllabus develop- 
ment proceeded. The SBOs from hierarchy and SBO development were 
organized and sequenced using the T and R syllabus as a general 
guide . 

An anticipated problem of any ISO program is FRS acceptance 
of the finished product. By staying within the general framework 
of the T and R syllabus, the probability of FRS rejection and/or 
requirements for major modifications, at least at the syllabus 
level of training course development, was reduced. This was a 
desirable by-product of the approach adopted rather than a goal. 
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The hierarchies and SBOs were mission oriented rather than 
training oriented. As a consequence many objectives which the 
SMLs considered essential for training were not included in the 
hierarchies. The ::iost significant subject matter area for which 
SBOs, were not contained in the hierarchies was systems information. 
Although training courses developed using ISD techniques do not 
generally include detailed systems information, it is still essential 
that certain background systems information, in addition to that 
which can be gleaned from the mission-oriented S30s, be taught. 
Other objectives which were not included in the h^^erarchies , but 
which are an important part of training, were aircraft maneuvers, 
such as touch and go landings and timed turns. Objectives to cover 
systems information and training maneuvers were included in lessons 
as the syllabus was developed. 

Throughout syllabus development, objectives not contained in 
the hierarchies were included in lessons. This is to be expected 
since ISD is a highly iterative process, i.e., as the process 
proceeds from task to task new insights are developed which improve 
the end products of previous tasks. As the SBOs were structured 
into lessons and the lessons were reviewed for completeness, 
objectives were added to provide complete instruction in certain 
topic areas. The addition of these supplemental objectives, plus 
the objectives for systems anv training maneuvers, will enhance 
the quality of the pilot and RIO training courses. 

Since media selection preceded syllabus development, in the 
ISD specification, it was implicit, although not directly stated, 
that media decisions should affect the grouping of objectives 
into lessons. This was ttue for the flight and simulator portions 
of the syllabus. The start point for these portions was a listing 
of all objectives which required simulatoir-s and/or flights. The 
organization of ground school lessons, however, was done on the 
basis of subject matter content and hierarchy structures without 
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regard for selected media, i.e., the overriding criterion for 
good lesson structure is appropriateness and completeness of 
content. 

Most of the ground school lessons contained objectives for 
which the sane medium had been selected. Other lessons, however, 
cantained more than one medium across the objectives. When this 
happened^ each lesson was examined to determine which medium was 
the most frequently selected for the objectives in that lesson. 
Media selected *or the remaining objectives in the lesson were 
then examined to determine if one of the alternate media from 
the media selection model was the same as the predominant medium 
in the lesson. In most cases, matches wev^e found and the pre- 
dominant medium was selected for all objectives in the lesson. 
In those instances in which no alternate medium matched, the 
content of the objective was examined to determine whether or 
not the objective should be repositioned in the syllabus. No 
objectives were repositioned. For each case the content and 
context requirements of the objective were judged to be more 
important than media considerations. 

Contract time allotted for syllabus development was only 
four weeks. This was an unreal istical ly short period for the 
amount of work required to structure pilot and RIO syllabi for 
both the Navy and Marines and to identify differences between the 
J and N aircraft models. In addition, SM£ support was less than 
that needed for the task. As a result, 14 weeks were required 
to complete syllabus development. 

Although validation of the syllabus by the three FRSs at 
which the training courses will be implemented was not contractually 
required, it was anticipated that this could be accomplished with* 
in the scope and time constraints of the contract. Due to the 
delay in ccr-: etion, however, it was not feasible to perform a 
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complete validation. Since lesson specification development 
started immediately after syllabus completion under another 
contract, it was decided that validation could be carried out 
concurrently with the early parts of lesson specification develop- 
ment. 

TRAINING SUPPORT REQUIREMENTS ANALYSIS. 

Overvi ew . The final task was to perform a Training Support Re- 
quirements Analysis (TSRA). The purpose of the TSRA w^s to 
estimate the costs of personnel, equipment, materials, and services 
which will be required to further design, develop, implement, and 
maintain the pilot and RIO training courses. This information will 
be used for future planning and budgeting. The detailed analysis 
is contained in the Training Support Requirements Report. 

During the design phase lesson specifications will be produced. 
Since this phase had been contracted prior lo the development of 
the TSRA, no cost estimates were made. Estimates for all other 
future phases of the F-4 ISO program were provided using the pro- 
cedures outlined in the following section. 

Procedure s . The devel opiiien t phase, which consists of the authoring 
and production of course mater ia 1 , is the single most expensive 
ph^se of an ISO program. Twi ^asic sources of information were 
used in the calculation of development cost estimates: (1) syllabus 
parameters on the number of lessons in each instructional medium 
and (2) personnel, materials, and service costs required to produce 
lessons in each med^jm. By merging these two sources of information, 
total estimated costs for production were determined. 

Syllabus parameters were determined by totaling the number of 
lessons in each medium. This data is shown in Table 5- Personnel, 
material, and service costs for the production of one lesson in 
each medium were ass ijiii la ted using the format shown in Figure 10. 
Appendix D contains development costs for each of the media selected 
for use in the F-4 training course. 
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INSTRUCTIONAL NO. OF 

MEDIUM LESSONS 

Ground School 

SI ide Tape 77 

Random Access Slide 12 

Workbook 2 

Videotape 1 

Medi ated I nteracti ve Lecture 3 

Simulator 89 

Aircraft 293 



Figure 5. Number of Lessons in Each 
Instructional Medium 
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Sunmiariz ing across categories within media provided total costs 
per medium. The total costs per rredium were then summed to give 
a-cost o<.Ljiiiat(j for the production of one complete set of course 
mater ia 1 s . 

Since it was understood uhat all reproduction of materials 
would be done-by the government, no reproduction costs were 
estimated.^ Estimates of the number of sets of consummable and 
non-consummabl e materials were provided to assist the government 
in estimating ^reproduction costs. 

In addition to production costs, -s^acilities requirements for 
the prodjction phase were estimated. Assuming pVoduction on-site 
at MCAS, Yuma, it was determined that the facilities available in 
the MCCTRG-10 ISD Department are sufficient to support the pro- 
duction ei^fort. Since it was assumed that any company conrractcd 
for production would furnish its own equipment, equipment costs 
for production were not estimated. 

The primary cost of •'"mpl er-en Lation will be the acquisition of 
carrels and components required for t^e presentation of slide- tape 
lessons. Determination of the number of carrels required at each 
Fleet Readiness Squadron (FRS') and each operational and reserve 
squadron thac will use the training materials was based on the 
number of lessons that will be presented via slide-tapes, the 
length of the training course, and the projected student flow 
rate. Th:: costs of fully equipped carrels were based upon current 
GSA prices for standard equipment purchased by the Navy and in- 
cluded a fa^^or to allow for inflation during the interval before 
the equipment is purchased. 

Facilities required to house the learning centers and to 
provide space for materials storage, lectures, and briefings were 
estimated. The results of a preliminary survey of the three FRSs 
indicated that spaces are available to satisfy the requirements of 
the new training courses; however, some modifications will be 
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required in setting up the learning centers. The<:e include in- 
stallation of carpeting, suppl eiiierrta 1 elecLrical power, and in- 
stallation of shelves for materials' storage. Due to the large 
number of potential users of the training materials in the active 
and reserve communities, it was not feasible to survey available 
facilities at the numerous locations. Since space requirements 
will be minimal, it is conjectured that adequate facilities are 
available at most locations. 

FRS instructor requirements for implementation were estimated. 
This estimate was based on the projected student flow rate through 
the training courses, the number of students that will be in 
training at any one time as derived from a schedule of overlapping 
classes, and the number and projected utilisation of media that 
require instructors- Included in the estimate was a factor to 
account for leave, TAD, etc. 

The final step in the TSRA was to estimate resource require- 
ments for five-year maintenance and evaluation of the training 
courses. Revision, mai irenance , and quality control activities 
will be most extensive during the first year of the five-year period 
Total requirements for the first year were dotonnined by multiplying 
the personnel, materials, and services directly tied to initial 
courseware development and production by .35. Annual requirements 
for the next four years were determined by applying a factor of 



The TSRA results indicated a costly development, implementation 
and maintenance program for the F-4 pilot and RIO courses. Total 
estimated costs for each phase were as follows: 



.20. 



Devel opment 
Implementation 
Five-year eval uati on 



$ 603,047 
118,266 



and maintenance 



Year 1 



Years 2-5 



211 ,066 
4 ^,436 



Grand Total 



51,414.815 
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The Training Support Requirements Report contained infor- 
mation sufficient to make cost trade-offs among the ground school 
media. Costs for each medium could be compared to determine 
relative costs. When used in conjunction with a graphical flow 
chart of the syllabus, which was part of the TSRA, the costs could 
be related to specific lessons, and po tial sources of savings 
could be identified. 

SME Trainin g. Since the TSRA was a contractor task, no formal 
training was conducted. A two-hour seminar was held to acquaint 
the SMEs with the contents, purposes, and appl icati ons of the TSRA. 

Problems and Solutions . Formulation of the TSRA went smoothly v/ith 



no major problerns. Several small probl emsJr however developed 



during the course of the analysis. These will be discussed below. 

Three production companies were surveyed to collect the data 
required to complete Figure 10. Among the companies there was a 
lack of consistency in interpretation of the roles and relative 
efforts of the different labor categories. This was particularly 
true for graphics specialist and paste-up artist. One company 
defined a paste-up artist as one who could also dc simple illus- 
trations, with the graphics specialist reqi^ired for complex illus- 
trations. Another company, however, assigned all illustrations to 
the graphics s.oeciali?- vith the oaste-up artist serving a 
supporting role for pt . t . -up work only. In addition to discrepancies 
in personnel roles and responsibilities, companies differed in hourly 
wage rates. This is due to different base rates across companies, 
as well as, different overhead rates. 

The values which were entered in the cells in Figure 10 and 
the wage rates used as multipliers for the personnel hours are 
averages across the three production companies. The estimates of 
materials development and revision costs are, therefore, considered 
to be representative of costs across the industry. Estimates from 
any given company would be expected to differ somewhat from those 
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in the TSRA, Overall, however, there was considerable agreement 
among companies, so that individual estimates should not differ 
radically across good production companies. 

During the formative stages of setting up a "learning center, 
NA, TRAEQUIPCEN media specialists perform a thorough analysis of 
facilities requirements- Included is the amount of space, as well 
as, the characteristics of the space. It was anticipated that such 
a preliminary analysis could be performed by Allen Corporation at 
the three FRSs and at several of the other user squadrons. This 
was not feasible since the squadrons could not precisely identify 
the facilities which will be used to house the audio-visual and 
supporting facilities. Instead, general guidelines were provided. 
These guidelines were based on requirements of the pilot and RIO 
courses and good learning center design, considered in terms of 
general characteristics of the types of facilities available at 
the different squadrons- A more thorough analysis will be required 
during the early portions of the development phase of the F-4 
ISO program. 

Based on information available after syllabus development, 
the TSRA is an accur- te estiniate of future resource requirements. 
The total estimates contained therein, however, may be subject to 
considerable update as a result of syllabus changes identified 
during les n specification development. As discussed above, ISO 
is an iterative prc^ess with subsequent insights affecting the 
end-products of previous tasks. It is anticipated that future 
changes will increase the size of the development effort rather 
than reduce it. The totals may change, however, the costs per 
lesson will not change. If there are resource constraints for 
development, the number of lessons contracted should be based on 
available or anticipated resources, with left-over lessons, if any, 

being taught in the same medium as in the current F-4 training 
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programs. The selection of lessons to "^e mediated should be 
\ based on the general characteristics of lesson content > such as 
^•update requirements, need for visuals, etc. The content of all 
lessons, however, should be as specified during the ISD program, 

TRAiNIvNG PROGRAM SUMMARY. 

The^ structure of the proposed training courses is embodied 
in ..-e pilat and RIO syllabi • The syllabi, rather than being 
^'concrete*' products, constitute a working document which will be 
revised during subsequent phases ov the ISD program. Due to the 
overlap of this contract with a subsequent contract for the 
development of lesson specifications, syllabus revisions are 
occurring during the [Performance period for this contract. Based 
on a preliminary evaluation of the revisions being incorporated 
by the SMEs, it is not feasi^ble to discuss in detail syllabus 
parameters such as length, nuniber of lessons per medium, etc. 
Rather, an ov.^rview in terms of^^media types, instructional 
strategies, and ut-lization of trainers and the aircraft will 
be provided. \ 

The candidate media were speci f ie&^^i n the media selection 
mode., thus the syllab'js contains only those media. They are 
mediated interactive lecture, workbook, slide- tape, videotape, 
simulator, and aircraft. 

The guidelines for syllabus development indicated that the 
number of flights should be maximized consistent with aircraft 
availability and projected student flow rate. The ISD^^^yl 1 abus 
contains essentially the same number of flights as the current 
syllabus. It is anticipated that this will not change duri^n^ 
revisions in subsequent phases of the program* 

The primary innovation in the proposed syllabus is the in- ^ 
di vidual i zation of instruction as opposed to the current 
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"lock step*', classroom orientation • The system is ind, calized 
in the sense that most ground school lessons are presented in 
either a slide-tape carrel or in a workbook with a test at the 
end of each lesson. Students can proceed at their own pace 
within lessons. Across lessons it is anticipated that self-pacing 
will be feasible within the constraints of equipment availability 
and prerequisite requirements. 

Self-pacing combined with the more efficient presenta*i:n of 
information in the individualized media should in theory yic^u a 
training program whose duration is less than that of non-mediated, 
paced programs. Due to external constraints such as aircraft and 
trainer availability and additional duties required of students, 
current projections are that this will not be the case for the 
new F-4 pilot and RIO courses. Typical com.pletion time for the 
Marine 60 percent portion and the Navy 80 percent portion should 
be about six months. Since the remaining 40 percent and 20 percent 
respectively are taught at ' operational squadrons where training is 
interspersed with normal operations, no estimate of duration for 
these portions can be made. 

The training courses outlined in the pilot and RIO syllabi 
adhere to the principles of good syllabus structure. That is, 
lessons are sequenced so as to optimize student learning and 
retention by integrating ground school instruction with "hands-on" 
practice in the trainers and the aircraft. Within lessons segments 
are organized to enhance instructional flow within content areas. 
It is anticipated that, when implemented, the pilot and RIO courses 
will produce better qualified, mere-standardized F-4 aircrewmen. 
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SECTION IV 
RESULTS AND CONCLUSIONS 

Results in terms of the structure of the pilot and RIO 
syllabi were presented in the previous section. In this section, 
results and conclusions relevant to the ISO model used in the 
program and to ISO in general will be presented under five different 
headings : 

ISD Methods 
ISD Imp! ementation 
ISD Resources 
Documentation Adjustments 
Speculative Observations 

ISD METHODS 

The ISD model used in this program is the most prescriptive 
model available in terms of the specification of discrete tasks 
required to produce an ISD-based training program. Within the 
tasks many of the procedures contain precisely delineated steps. 
The goal of "mechanizing'* ISD into a straight-forward set of pro- 
cedures which can be executed by "turning the crank", has been 
pursued by training technologists for many years. In the 5-2 
model this goal has been, at least, partially achieved. 

Since the procedures detailed in the niodol are highly structured 
relative to other approaches in which broad guidelines rather than 
detailed procedures are provided, it is easy for a training tech- 
nologist first applying the model to implement the procedures 
without considering the intatigible, judgment factors which are 
integral components of ISD- This is not a criticism of the goal 
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of procedural 121 ng ISD. Rather, it is a caution against taking 
the procedures at face value without considering the rationale 
behind tne steps. In studying the procedures in light of general 
principles of learning and real world constraints^ insights will 
be gained which assist in integrating the intangible factors into 
training program development. Also, additional general procedures 
and others which are particularly relevant to the system under 
scudy will be derived. 

In the following paragraphs, conclusions and recommendations 
pe>"taining to each major task carried out in the F-4 ISD program 
will be discussed. Some of the points were considered in Section 
II, Implementation, while others are presented for the first time 
in this section. 

OBJECTIVES HIERARCHIES AND SBOs . The task of developing objectives 
hierarchies and SBOs went smoothly. The SMEs understood the tech- 
niques and implemented them well. One deficiency in the procedures 
outlined in the model became evident during the development process. 
This was the lack of clearly defined guidelines for determining 
the level in the hierarchies above which d*^" ^'1 was not sufficient 
and below which additional detail was trivial. The source of the 
deficiency in the specification was the failure to relate objectives 
to the way in which they will be used during subsequent ISD tasks, 
i.e., as segments of instruction. In order to remedy the deficiency 
it is recommended that a description of how objectives will be 
converted into segments of instruction be included in the instructions 
for hierarchy and SBO development. As part of this description., 
guidelines should be provided which define the scope of a segment 
of instruction, e.g., two to five miriuLc^s of ground school in- 
struction for which four to eight supporting visuals will be 
required; for iiianipul ati ve tasks, an action which can be evaluated 
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arid would be expected to be practiced from tw r to five minutes 
during a simulator or flight lesson. By provi such guidelines 

'*"he level of det:^ ! in the hierarchies will be lore uniform. 
More importantly, however, this consistency will contribute to 
more efficient syllabus or^gani zati on and to less revision during 
the development of lessen specifications. 

The examples presented above are not intended to be specific 
guidelines. They are based on limited experience gained during 
the early stages of lesson specification development. Data from 
this and other programs should be gathered and analyzed to deter- 
mine appropriate parameters which define a typical segment of 
instruction . 

The conditions and standards portions of each objective 
serve no function during hierarchy development. They may serve 
some function during media selection, given a selection model 
into which they are incorporated. At this point in ISD model 
formulation, it is not clear how they would be used for media 
selection. Certainly the model specified for the F-4 ISD program 
did not utilize conditions and standards. 

Tc formulate conditions and standards well is time consuming 
for the SMEs and detracts from the primary task of identifying 
and structuring training objectives. Even well -thought out con- 
ditions and standards are not optimal at the hierarchy stage of 
training system development. Such conditions and standards can 
be specified only after media have been selected and testing 
philosophies have been established. In the present ISD model 
media selection follows hierarchy development, with the formulation 
of testing philosophies taking place during lesson specification 
development. It is recormiended that the requirements for 
conditions and stcindards during SBO development be eliminated. 
This will allow hierarchy development to proceed more quickly and 
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v/i 1 1 allow the SMEs to concentrate on their expertise, knowledge 
of the aircrdfi. Dctai lea conditions ^r\d ^"tandards would continue 
tc be formulated as part cf lesson s'.ec if i ca ti ons as outlined in 
the ISO model • 

A primary deficiency in the F-4 hierarchies is the lack cf 
objectives which cover systems information and training-only objectives. 
This resulted from the mission orientation cf the task "^^'sting. 
When the task listing was expanded into hierarchies and SBOs, the 
mission orientation was maintained. Some systems information^ 
particulary erre^^genc ies , is contained in the hierarchies; however, 
the amount and detail of the information is significantly less 
than that which is acceptable tc the r-4 training corrjnunity. To 
remedy this situation systems objectives were added during syllabus 
development. Efforts were made to insure that tre supplemental 
objectives were relevant to mission performance and were not 
tangential from an ISD point of view, but were sufficiently de- 
tailed to satisfy the training community. 

Training objectives were also added during syllabus development. 
These were primarily performance objectives which are required in 
the training environment but which would not be reflected in 
mission-oriented SBOs . Examples of these objectives are touch 
and go landings and timed turns. 

It is recommended that systems and training-only objectives be 
identified during the development of hierarchies and SBOs. Most 
of these objectives can be incorporated into the hierarchies as 
prerequisites to objectives identified as mission objectives. 
Others may not clearly relate to the mission objectives and will, 
therefore, constitute a second category of objectives. The 
fundamental concern is that the identi f i ca ti or> and structuring of 
objectives be as complete and accurate as possible, at as early 
a stage in the ISD process as possible. 
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MEDIA SELECTION. The media selection model specified for use in 
the F-4 ISD proc;ram is not required in thr ,iost recent specifi- 
cation of training requirements for aviation weapons systems^ 
MIL-T-29053. Rather, the contractor is given the latitude to 
select a model v/ithin certain constraints, subjecc to contract 
technical monitor approval. The current mode! is presented as 
one alternative which satisfies the specified c^ns-^aints. This 
latitude afforded future ISD contractors provides an indication 
that NAVTRAEQUIPCEN ISD personnel view media selection as a part 
of the ISD model which needs improvement. 

This contractor has not formulated a media selection model 
which is any more prescriptive than the F-4 .model or the numerous 
other models described in the literature- There are^ however, 
several recommendations which will improve the model used in the 
F-4 program. More precise definitions of the questions and 
responses need to be included in the description of how to use 
the model. These definitions should be accompanied by a number 
of worked examples which span the range of types of objectives 
which will be traced through the media map. 

The number of candidate media identified in the model should 
be broadened to reflect other feasible devices which are available. 
Since s e media are similar in terms of instructional capabilities, 
these can be grouped into media classes with iurther differentiation 
based on cost and dependability. 

If the F-4 media model is used ]n future programs, it is 
recommended that delivery of multiple copies of a completed media 
map for each objective not be required. Since media selection is 
altered somewhat du^^-'ng syllabus development and since the size 
of deliverable voluu.es is prohibitive^ media selection should be 
considered an interim step not requiring a formal deliverable 
end-product. Media decisions v/i 11 be summarized after the syllabus 
has been developed. 
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Finally, media selection should be a part of syllabus 
development rather than a preceding step. Performance objectives 
need tc be identified so that simulator and flight lessons can 
be organized. Ground school media, however, need not be selected 
until after the ground school lessons have been organized- These 
lessons should be structured on the basis of subject matter content, 
not on the basis of iT.edia selected for each objective. This pro- 
cedure will elimir^iate the duplication cf effort in which media are 
selected for each objective but some of which are changed as a 
result of syllabus organization. 

SYLLABUS. The guidelines for syllabus development provided in the 
specification are good. They are idealized and cannot always be 
strictly followed due to external constraints, such as training 
policies and simulator capabilities. They are, however, quite 
workable within the unique constraints of different training situations. 

Several recommendations have resulted from use of the syllabus 
development procedures in the F-4 ISD program. The specification 
that the syllabus should contain the maximum feasible number of 
flights is not in accord with traditional ISD philosophy. In 
most cases the maximum feasible number of flights will be the 
number in the existing syllabus. One goal of ISD is to decrease 
flight time while increasing the quality of training provided. 
The number of flights should be based upon the number of objectives 
which require flight instruction and practice, and the amount of 
practice required to master the objectives. Admittedly, this 
guideline is not stated in operational terms such as the existing 
guidelines. It is, however, more realistic a-^d consistent with 
the goals of ISD. 

The specification states that flights should proceed from 
"easier to more difficult" by adding new objecn^ves to each flight. 
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Taken literally this statement can be misleading. As new objectives 
are added, flights become more complex, but not necessarily more 
difficult. Students may have mastered earlier objectives so that 
the addition . Q-^^ new objectives dees not increase difficulty. It 
is recommended that '^easier to more difficult'* be changed to 
"simpler to more complex-" In addition, it should be recognized 
that within the total syllabus, a flight to flight increase in 
complexity may not occur, even though the general flow is from 
simpler 10 more complex* For example, in the F-4 pilot syllabus, 
air-to-air training precedes ai r^ to-g "cund training. The final 
flight in th^ air-to-air portion of the syllabus is "more complex" 
than the first flight the air-to-ground portion. 

One of the syllabus guidelines states that a flight lesson 
shall be the last lesson in a unit which contains a flight and 
that the next lesson is by definition the first lesson in the 
next unit- This is often the ca^c. In other instances, however, 
iz is more logical for units to contain multiple flights if the 
objectives to be pract'ced in the flights are essentially the same 
and if there is no ground school or sii.iulator instruction separating 
the flights. If subject matter content is uniform, including 
multiple flights in the same unit when there is intervening ground- 
based instruction should also be allowed. This will enhance 
continuity within units and provide more flexibility to structure 
units of approximately equal size. 

The problem in syllabus development encountered in this pro- 
gram (See Section II, Implementation) did not result from a basic 
deficiency in the specification being followed. The process of 
developing a syllabus requires a large judgment component v/ithin 
the specified development guidelines. Judgment varies across 
individuals, thus, the SMEs had difficulty agreeing on the best 
way to structure the syllabus. This is to be expected in any 
ISD program, particularly in one for a system which has been in the 
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inventory for a number of years. 

TRAINING SUPPORT REQUIREMENTS ANALYSIS. The format provided in the 
specification for collecting production personnel and cost data 
was very useful The personnel, materials, and services catenories 
were inclusive for the media specified in the media selection model. 
Other costs, however » were not specified in sufficient detail or 
were not mentioned at all. Also, the funding sources for the 
different costs need to be identified^ This enables contracting 
personnel to differentiate between costs which will come frow. ISD 
resources and those which will come from othe^ resources. 

EqL.ipment categories should be more precisely identified. 
This will ass'ist in nore accurate and complete costing and will 
allow for distinctions between government-furnished and contractor- 
furnished equipment. For example, categories of equipment required 
during production include materials production gear, reproduction, 
and typewriters and composers. Similarly, a breakdown of 
facilities in terms of intended use is desirable. 

In the specification, no cost category is included for the 
formulation and production of implementation and quality control 
plans and an instructor training course- These are important 
parts of the production phase and should be included in the pro- 
duction cost estimates. Also, costs incurred by the contracting 
aqency should be included. 

Revision estimates for the first year after implementation are 
determined by applying a factor of .36 to materials production costs 
A factor of -20 is used for each of the other out years. The 
limited data from other ISD programs suggests that a more accurate 
factor may be .20 for the first year and some lesser factor f l r 
subsequent years. More data needs to be collected to determine 
the appropriate factors to be used in revision cost t^scimates. 
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ISD^ IMPLEMENTATION 

Implementation of the ISD model was successful and occilrred 
as planned with only minor deviations as discussed in other sections 
of this report. There are, however^ three general issues cdncerning 
the manner in which ISD is implemented. The first issue, contractor 
versus military SMEs-^'did not arise during this program. The last 
two issues,, the quantity a'nd quality of ntilitary SME support and 
'end product acceptance by the draining community were issuer raised 
in" this program. All three issues will be discussed below. 

SOURCE OF SMEs. .Other ISD programs have used contractor SMEs 
exclusively, military SMEs exclusively, or combinations of both. 
Arguments for the two extremes^ essential ly boil down to quantity 
and quality of work produced versus latest system knowledge and 
program acceptance. The limited data available indicate that 
contractor SMEs can be expected to produce more and better end 
products than military SMEs in a' given time period. On the other 
hand, military SMEs are more likely to be versed in current system 
operating procedures and tactttrs. In addition, training programs 
produced by military SMEs are more likely to be accepted than 
those produced by contractor SMEs, i.e./ if the producer and user 
are one "in the same, there is a higher probability of acceptance. 

The selection of- an option ' cannot be based solely on a trade- 
off of the above factors. Also to b^ considered are qualifications 
of military SMEs who will be assigned, the level of corrtraitment 
once assigned, and the ISD phase to which they will be assigned. 
The former two considerations will- be discussed in the following 
section on the structuring of a team of military SMEs,. 

The effect of ISD phases is a function ^of the different 
types of input reqaired iuring the different phases. The basic 
structure and subject ma:ter of the training program is determined 
during the analysis and design phases. In order to insure thot 
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the outputs of tiiese phases are technically correct and are 
acceptable to the trciining conmunity, it is important that 
nn'litary SMEs be assigned. They may be supplemented by con- 
"iractor SMEs to improve productivity, but must be heavily in- 
vo'. ved. 

During development of training materials a heavier emphasis 
may be placed on the use of contractor SMEs/authors , who are 
wel 1 -schooled i produccion procedures. The cont^^actor SMEs 
will refine the products of the earlier phases but will make 
no suDStantive changes without input from military SMEs, By 
recognizing this division the capabilities of the different 
personnel will be more effectively utilized. 

ASSIGNMENT OF SMEs. It is desirable for SMEs assigned to ISD 
progroiiis to be instructors in a training squadron or to have had 
recent instructor cxpey'icncc. Thos personnel can provide 
greater knowlecye of squadron training policies and general training 
techniques than can those without instructor experience. In 
addition, their close association with the tr-ining community 
enhances the probability of squadron acceptance* 

A recurring problem in ISD programs is the failure to assign 
enough SMEs or to assign SMEs whose other duties significantly 
deti^act from their contribution to the ISD program. Ideally, an 
"SD departirient snculd be formed with.n the training squadron. 
> £s sho'jld be assigned full time to the program with other duties 
^c^":uced *:o an a::so' ' ^: mi n iniuir.. Assignment to ISD should be on 
a 'c^g teriT: ta to m aintain stability and cont nuity within the 
* tearr. "his would reduce turnover and the resultant in-- 
--^-^^ : c : e- : ^ e- whicK' nave caused L-roblens in previous ISD programs. 
^ : ISZ deoc'^t.-i^^t must have a sirong leader who is 
■ ' vc tc : - reaj^" ^ements and who is willing to work 

- . ^.'^'t.^. t'^.'r c^rtr^dct^ ^. T ince ISD is relatively new ar^i 

■-'^t". :^-,a : '^ear: of tra^'ning c^ctiv-^ty, there is a 
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tendency for looseness in the ISO team. This leads to wdrk in- 
efficiencies, lack of interest, and waste of time- 

A related problem is the status accorded ISD and the SMEs 
involved in an ISD program, in general, ISD assignments are not 
considered career-enhancing and are not given as rewards for a 
good record of military performance or good performance in the 
weapons sytem. Such a situation can lead to morale problems and 
low productivity. 

A sumnary of considerations to be implemented in the formation 
of military SMEs into an ISD team, the structuring of the team, and 
the status of the team is as follows: 

^ Make the -ISD team an integral part of the training 
department rather than a separate entity, 

° Assign SMEs who are current instru.tors in the 
training squadron. 

° Make SME assignments on the basis of merit witn 
the clear understar^'^ that participation in the 
ISD progrcm is a or -a which will allow those 

assigned to have a .ye impact on the future of 
the weapons system ining program. 

^ Select a team leader wkO is responsive to ISD re- 
quirements and who will provide strong direction 
to the ISD team while working closely with con- 
tractor special -^'sts. 

^ Assign SMEs "" ^1 time with extra duties kept to a 
m inimum, 

° Select SMEs who are exoected to remain at the duty 
station for the duratior of the ISD program. 

° Assign a ful i compUment of SMEs based on require- 
ments from previous similar ISD programs and 
manpower projections for ihe current program. 

Rele :ve to other ISD programs, overall SME support for the 
F-4 progran. /;as good. The ISD de.iartment was created and staffed 
with four Marine SMEs prior to contract award. Initial selection 
and organization of SMEs v.^as, therefore, not required. Despite 
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Lhj advantage of early organization there were some problems with 
SME suppr-^t. These problems were as follows: 

1. There was insufficient participation by iNavy SMEs. 
Navy SMEs from VF-121, NAS , Mir-'^iar, were assigned on 
an intermittent basis with no SME participating more 
than 45 days. This situation caused a lack of con- 
tinuity in Navy input and complicated the process of 
identifying Navy/Marine differences . 

2. Marine SMEs were assigned other duties, such as flights 
and simulator lessons. This significantly reduced the 
amount of time devoted to ISD work. 

3- In addition to time lost due to other duties, t^ere 
were inefficiencies caused by a lack of motivation, 
usually uh i b wab a crarib i en L problem. During syllabus 
development, how^ /er, this became a significant problem 
which ultimatel caused the program to fall behind 
schedul e. 

4. The ISD department was "^lOt part of the FRS . Although 
this organizational structure did no L significantly 
hamper the program, it did effect communication between 
the SMEs and FRS training personnel, i.e.r members of 
the training squadron did not as freely interact with 
the SMEs as was desired. It appeared that the' ISD 
SME5 were considered "outsiders" by the instructors. 

5. The nurrirer of SMEs assigned (4) was less than that 
specified in the F*4 master plan (6 Marine, 2 Navy). 
Although the four SMEs assigned, supplemented by Navy 
SMEs as discussed above, generally ^erformed well, 
additional SMEs would have improved program progress 
and qua", ity . 

ISD PROGRAM ACCEPTANCE. The third major iss..e to be discussed is 
the readir.'':ss of the Navy training communities to accept Lraining 



84 



NAVTRAEQUIPCEN 77-C-0081-1 



programs produced through the use of ISD. This issue has two 
facets: (1) the reluctance to change the way "it has always been 
done-" This is particularly true for a weapons sytem, such as the 

which has been in the inventory for many years and for which 
training doctrine has been precisely defined; (2) the tendency 
for the ISD program to degrade over time with a return to earlier 
training techniques . 

To help insure that ISD training programs will be accepted 
and implemented as developed, the ISD procedjres must be carried 
out within the franiework of existing training policy, i.e,, working 
through the ISD procedures per se^, does not insure viabili^" 
of the end product. It must be remembered that the inertia of any 
well-establ ished system works ageinst changes. Change - ^t, 
therefore, be closely worked out with those who perp^.uate the 
inertia. In the case of an aircrew training system, curreht 
training personnel, as well as policies from higher authority 
"re the primary sources of inertia. 

In applying ISD technology to an existing weapons sytem, ISU 
specialists must work closely with training personnel. In doing 
so, the development process becomes one of "give and take-** Some 
aspects of the emerging training program, which are formulated 
using ISD procedures, will be unacceptable to the training community 
or will counter existing rigid policies. These aspects must be 
modified to achieve c^cceptance. On the other hand, existing 
training procedures may be better than or enhance training pro- 
cedures resulting from an ISD analysis. In this case, the output^ 
of the ISD analysis will be upgraded. Although intangible and 
not definable in concrete procedural steps, this high degree of 
interaction during training program development is a primary 
key to ultimate success. 
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Once an ISO program has been implemented, the ISD effort 
has not ended. On the contrary v a new phase of ISD has begun. 
During the remainder of the life cycle of the weapons system, 
the ISD-based training program must be changed and updated as 
the weapons system changes and as data on the effectiveness of 
the training program is acc\:mulated. To do this a nucleus of 
training personnel schooled in ISD pro^cedures and knowledge in 
the training program must be maintained. In addition, resources 
must be provided to produce and implement changes as they are 
identified. Without a continued emphasis on ISD, even the best- 
formu'^^ted training program will fall into disarray. 

As ISD procedures continue to be refined, a mechanism is - 
being formulated to cope with the acceptance issue. This in- 
volves structuring ISD programs so that all relevant parties are 
included from program inception through implementation and main- 
tenance. Relevant parties range from CNO down through the user 
squadrons and include high level commanders, government educational 
specialists, facilities and equipment specialists, contracting 
agencies, contractors, FRS instructors, and procurement personnel. 
This structure will provide command emphasis to ISD programs 
and will assist in coordinating the activities of the many agencies 
invol ved. 

■eneral accepta.ice of ISD will be a gradual process. The 
organizational concept discussed above will facilitate the process. 
The biggest boost for ISD, however, will be a history of successful 
orograms. 

ISO RESOURCES 

The main resource used in this project was labor. Labor 
expended by category and task is summarized in Section V, Resource 
Utilization. These -^^anhour figures are accurate as will be 
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discussed below, however, th figures for some of the tasks do 
not provide good guidelines for future planning purposes. 

Since the development of objectives hierarchies and SBOs 
began prior to contract award, manhour expenditures for this 
task do not indicate the labor required to perfcrn) the entire 
task. Records maintained by the previous contractor for the 
initial portion of objectives hierarchy and SBO development we. e 
incomplete and were not included. 

About 50 percent of the hierarchies were developed under Allen 
Corporation direction. Existing hierarchies were revie ed by 
the SMEs and Allen personnel and in many cases were revised. It 
is estimated that the total SME time required to learn the pro- 
-^edures and to produce the hierarchies and SBOs was 600 manhours. 

1 additional 100-200 professional manhours was expended by the 
previous contractor in the early stages of hierarchy development. 

Manhours required for media selection are good estimates for 
future planning, given use of the same selection model. The 
revised 6.2 Specification allows latitude in the choice of a media 
selection model. In future ISD programs, therefore, other models 
may be used which require mor^ or less effort than the one used 
in this program. 

SME support was "decking during syllabus development. This 
caused more calenda :ime than anticipated to be spent on this 
task. The increasec calendar time led to the expenditures of 
more contractor professiona"! manhours than would have been re- 
quired if SME support had been ample. 

In a previous section, several suggestions for streamlining 
the model were presented. In doing so, certain efficiencies would 
be gained in irr^plementing ISD; however, it is difficult to 
estimate the amount which would be gained. The labor expendi- 
tures for the four primary contract tasks, plus project ddmi ni strat i on 



87 

so 



NAVTRAEQUIPCLN 77-C-0081-1 



and reporting, are considered to be about the minimum for an JSD 
program of this size. It is projected that the increased 
efficiency resulting from streamlining the model would increase 
the quality of the products rather than significantly reduce 
manpower requi rements . 

Another significant resource was that required to prepare 
and duplicate the many deliverable end-products specified in the 
contract. Clearly, monthly progress reports and the final report 
draft and final versions are necessary to enable tne contract 
technical monitor to track progress and to get an overview of 
■project activities. It is recommended that, requ": rements for the 
submittal of the other major end products (Hierarchy Report, 
Medi4 Report, Syllabus Report, and TSRA) be modified in future 
contracts. These recommendations are as follows: 

1. Delete the Media Report and provide a summary media 
selections in the Syllabus Report. 

2. Require final versions only rather than draf. and final 
versions of each of the four documents. 

3. If the Phase I contract includes lesson specifications, 
. make the Syllabus Report a working document subject 

to review by the contract technical fwOnitor with the 
syllabus included in the Lesson Specification Report. 
This v;i 1 1 eliminate the need for Syllabus Update 
Reports during lesson specification development. 
Since the present cc tract did not include lesson 
specifications, this recommendation is not pertinent 
to F-4 ISO. In other ISD programs, however. Phase 
I has included lesson specifications. 
The net result of these recomn^endations for Phase I activities 
is that one. version of the Hierarchy Report, Syllabus Report, and 
TSRA will be required. The Hierarchy Report contains the data 
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base for the remainder of training program development and should 
be submitted foi review, as well as being provided in multiple 
working copies for use in subsequent phases of tho program. The 
Syllabus Report and TSRA are the tinal products prior to lesson 
specifications. If Phase I terminates with these products^ rathe 
than continuing through lesson specifications, both are requ . r ed 
as wrap-ups to Phase I activities, 

DOCUMENTATION ADJUSTMENTS 

In the discussion of ISO Methods in this section rec mendat 
for changes to the ISD specification were presented- These 
recommendations are suii:r:ari zed bel ow: 

1, In the discussion cf procedures used in hierarchy 

development provide better guidelines for determining 
how deep a hierarch should be. :t is suggested that 
in addition to the current guidelines the lowest 
appropriate level in a hierarchy be c?fined in terms 
of the amount of instruction required to teajn a base 
level objective in ground school. 
2- Delete conditions and standards as requirements for SBJs. 

3. Augment instructions for hierarchies and SBOs so that 
when developed the hierarchies and SBOs contain svstems 
and training-sijeci f ic objecti\^es, as well as, mission 
objectives. The systems and training-specific objectives 
can be included within the framework or the mission 
objectives without altering t .e ^"ission orientation. 

4. Incorporate media selection into syllabus development 
and select media by lesson rather than by ob.'iective. 
This would eliminate the need for a separate ^edia 
report and may require ref orm::! a ti on of the selection 
mode 1 
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5. Restate syllabus guidelines as follows: 

a. Base the number of flights on the numDer of flight 
objectives and amount of practice required for 
each rather than specifically specifying the 
maximum feasible number of flights. 

b* Sequence flights from "simple to more co, -plex 
rather than from "easier to more difficult-' 

c. Structu. J units on the basis of content rather 
than specifying that a flight is the last lessen 
of any unit that contains a flight. This will 
allow multiple fl ights containing the same or very 
similar objectives to be included in the same unit. 

6. Expand zhe guidelines for the TSRA as follows: 

a. Identify the government funding sources for different 
costs. This will help contracting personnel determine 
which costs will come from ISD funds and which wiTl 
come from other sources. 

b- Further define equipment and facilities categories 
in detail comparable to that provided for materials 
producti on . 

c. Include costs for technical work and documentation 
required under the 5,2 specification, such as 
implementation and quality control plans and an 
instructor training course. 

d. Consider revising the multiplication factors used 
to estimate revision costs. 

e- Sreak revision costs down so at costs for evalu- 
ation and revision specifications .-Bre distinct 
from costs for materials product;Kon curing revision. 



f. Sum manhour requirements to derive the number of 
personnel required in each ''labor category. 

7. Increase the detail in the Dli) for the final report, so that 

the general scope of each section is well-defined. Currently, many 

of the section Headings do not clearly indicate the general content 

areas expected to be covered. Another approach is to make the DID 

more general, thus allowing contractors greater flexibility in 

formulating the report. 
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SPECULATIVE OBSERVATIONS 

A great deal was accomplished during the first phase of the 
F-4 ISD program. A detailed, prescriptive ISD model was studied, 
interpreted, and implemented by a contractor not involved in the 
formulation of the model. Although certain deficiencies were 
identified a result of experience during implementation, the 
model essentially stood on its own and provided definitive 
guidance through the sequence of ISD tasks. The viability of 
the model was not challenged. Recommendat^ions for upgrading the 
model, as well as, more general ISD considerations were presented 
in previous sections. Several additional points will be discussed 
below. 

A large data base was collected and manipulated in this pro- 
gram. At times there were information management problems- due 
to the size of the d"^^ "--^e. A possible solution to this problem 
is the use of a coinpucerized system which contains algorithms for 
production of the data base (i^e., guides SMEs through hierarchy 
and SBO development) and for manipulation of the data base (e.g., 
assists in organizing and sequencing objectives into a syllabus). 
It is un '^rstoo:; that computer routines for this purpose have 
been or are ct^^rently being developed. ' Th^ application of these 
routines "o" aircrew 'SD programs should be studied. 

In this report and reports on other ISD programs, considerable 
discussion has been devoted to the quality, source, and number of 
SMEs participating in ISD programs. Very little discMSsior, 
however, has been devoted to the composition of the contractor ISD 
tean. Particularly relevant are the qual if ice "lions of the 
professional personnel who work closely with the SMEs in producing 
the various ""^0 end-oroducts . 

The guidelines provided by NAVTRAEQUIPCEN for the selection 
jf personne' emphasize educational and training application. 
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Clearly these are important criteria, particularly for the progrcim 
manavjer. To supplement the more '.heoretical training personnel an 
invaluable asset to the contractor team is a person{s) knowledge- 
able in military training doctrine and in the weapons sysiem under 
study or a similar system (i.e., a retired or discharged SME). This 
type of person is able to relate to the military SMEs and to solicit 
input from the SMEs that otherwise may be overlooked- The ability 
of cc.itractor personnel to relate to military personnel is an in- 
tang-»'ble. yet very importarit, component of a successful implemen- 
Ldtion of ISD i)rf)ctHlijri-. . In ciddition, ho is able to review subject 
nuitL'^r LonLcnL tor cDnip I L^eMiess tirui dccurMcy, thus complimenting 
other reviews for adherence to ISD s[)ec i f i ca t i ons . 

One final broad consideratior, is the readiness of the Navy 
training communities to accept ISD-based training programs. At the 
present they are not ready to accept ISD in its purest form. The 
question then arises whether ISD needs to change or the training 
communities need to change. The ansv/er is that both need to adjust, 
ISD to military doctrine which has evolved over cime'and much of 
which is sound, and the training communities to the benefits 
inherent in a thorough, systematic tr^iining analysis. Current 
training can nnd is being improved through the applicatfon of ISD, 
but only to the degree tnat ISD is Vesponsive to the n/eds of the 
training communities. Increased acceptance^of ISD wi/i come as 
in-seryice ISD capabilities are developed and as the structure 
for adipini straticn and control of ISD programs is refined. 
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SECTION V 
RESOURCE UTILIZATION 

I'his section presents an account of the resources used during 
the F- l/r. ISD progr^am. There are separate categories for personnel, 
time, and facilities and equipment. The purpose for collecting 
and summarizing resource data is to expand upon like data collected 
by NAVTRAEQUIPCEN from other ISD programs. This data can be used 
to establish a reliable basis for planning i.-; future ISD applications. 
More specifically, the data contained herein provides resource 
requirements for implementation of the 6.2 ISD model, Sinco this 
model is the one currently being refined as a standard for future 
Navy ISD programs, resource recjui remen ts from the F-4 program are 
particularly relevant for future planning, 

PERSONNEL 

Principal on-site technical contributors to the F-4 ISD program 
were an Instructional Psychologist (IP), who orcvided instruction 
and guidance in ISD techniques, a Production Mar.ci^er (PM), who co- 
ordinated ISD and administrative activities, and Subject Matter 
Experts (SMEs), Four SMEs were initially assigned to the program. 
The number of SMEs actually working on vhc- proc ram varied throughout, 
ranging from three to six. The number of manhours expended for each 
personnel category by ISD step is shown in Table 6, 

^ABLE 6. MANHOURS EXPENDED BY ON-SITE 



ISD 


TECHNICAL 


f^ERSONNEL 




ISD STEP 


IP 


PM 


SMEs 


Hierarchies and SBOs 


124 


250 


200 


Med 1 a Se'' ?r " i on 


115 


85 


42 


Syl 1 abu' 


385 


300 


401 


Training Support 
Requirements Analysis 


7.5 


15 


0 


TOTAL 


699 


650 


643 
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The data presented in Table 6 reflect actual time spent on each 
task. Other tasks, such as project administration, preparation of 
progress reports and memos, etc., are not shown. 

At the time of contract award to Allen Corporation, the 
develppment of objectives hierarchies and SBOs was approximately 
50 percent complete. The manhour totals for this task in Table 5 are 
therefore, estimated to ^be about half those required to develop 
the full set of hierarcr, es and SBOs for the F-4. 

The total on-site contractor p-^rsonnel hours are snjwn in 
"^^^-^lo 7. These figures contain the totals from Table 6 plus 
manhours for secretarial and administrative functions. 

TABLt 7. TOTAL PROJECT-RELATED MANHOURS 

SPENT BY ON-SITE CONTRACTOR PERSONNEL 



STAFF KANHOURS 

{ nstJ^ucti onal Psychologist 904 

Production Manager 960 

Administrative Assistant/Secretary 1,063 



Total nours of off-site [jersonnel by category are shown in 
Table The categories are teci^nical, management, and support. 

Technical hours include assistance with and review of on-site 
activities and preparation of reports. Management hours include 
contract managemen^^SSprganization and review of contract deli^er- 
c-bies, and lidTsrn with the contract technical monitor. Support 
hours ircluc^' typing and general secretarial assistance in support 
of off-site technical and manc'^ement efforts. 



PERSONNEL CATEGORY .MANHOURS 

Technical 40 

I^anagement 575 

Support 185 
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The overall .^a -; Jvsr expenditures were in accord with those 
oriqinally projected; however^ individual task requirements 
differed from the original projections* Since the development of 
objectives hierarchies and SBOs began prior to contract award, 
completion of this task required fewer manhours than projected . 
Media selection also reauired less time than projected. Syllabus 
development was a difficult task. The difficulty of the task 
combined with a shortage of SME support during this portion of 
the contract caused more contractor hours to be used. As a result, 
the milestone for syllabus completion was not met- This will be 
discussed further in the next section. Finally, the Training 
Support Requirements Analysis was completed as projected. 

TIME 

A summary of the projected versu^- actual elepsed time for each 
ISO step, is shown in Figure 11^ The program was on or ahead of 
schedule through media selection. As discussed in the 'Problems 
and Solutions" section under "Syllabus Development," problems were 
encountered which precluded the completion of syllatius development 
as scheduled- These problenis were the brief ainounL of tin»e allotted 
for syllabus development, the inherent difficulty of the task, the 
lack of agreement among SMEs, a lack of adequate SME support, and 
the multiple syllabi which had to be developed. Failure to coFr^plete 
the syllabi on or near schedule caused the Training Support Re- 
quirements Analysis (TSRA) and the Fina^ Report to be delayed. 

At toe outset it was .recognized that the schedule requirements 
for this program were demanding. Tin<-?s allotted to the four major 
task's were: hierarchies and SBOs (including SME training) - 11 Wv^eks; 
mec a selection - 4 weeks; syllabus development - 4 weeks; and 
TSRA - 4 weeks. Four weeks euch for media selection and t'^e 
training support requirements analysis were sufficient to conciete 
the tasks. As di scussed above , the four weeks scheduled fnr 
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syllabus development was not sufficient for a program of this 
size in which multiple syllabi are required. It is recommended 
that in similar future ISD programs, a minimum of ten weeks be 
scheduled for this task. With adequate SME support, it is 
Hypothesized the syllabi required in the F-4 ISD program could 
have been completed in ten weeks. 

The objectives hierarchies and SBOs were completed on 
schedule; however, this was partly a function of their having been 
started prior to contract award. For future [planning purposes^ 
it is recomniended that in programs of this siie, in which SMEs 
are the primary developers » a minimum of 15 weeks be allowed for 
SME training and the development of objectives hierarchies and 
SBOs. 

FACILITIES AND EQUIPMENT 

All -'^cilities and equipment were government furnished and 
were adequa^'.e for the program. The four assigned SMEs from 
MCCTRG-10 occupied their normal workspaces in two offices. A 
large conference room was avai'^able for use by SMEs who supplemented 
the four permanently assigned and for group meetings, training 
sessions, etc. Two offices with desks were^ provided for contractor 
personnel. One was used by technical personnel. The other was 
used by the administrative assistant/secretary and also contained 
filing cabinets and a work table. All rooms adjoined thus enhancing 
interaction among all personnel involved in the program. 

Reproduction capability was marginal when contract work began. 
During the contract, however, this capability was upgraded by the 
government and was quite sa .actory for the remainder of the 
contract. Government furnished typewriters were IBM Mag Card lis, 
which significantly facilitated production of the numerous large 
documents required under the contract. 

It should be noted that the facilities and equipment were a 
very positive factor for both the SMEs and contractor personnel. 
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They provided a good working environment both in terms of space 
and capabilities. 

The contractor provided consumable supplies and a telephone 
on-site. All off-site activities took place in contractor 
facilities. Since end products of the four major tasks were pro- 
duced on-site^ off-site activities primarily involved review, 
accounting, and nianaqement functions for which few resources were 
requi red. 
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APPENDIX A 



STRUCTURE OF 



JLCTiVES HIERARCHIES 



This appendix is an extract with minor modifications 
from one section of the Objectives Hierarchy Report 
{CDRL Item EGOS), It presents a description of the organi- 
zation of the hierarchies, the numbering system, and the 
symbols used. 
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STRUCTURE OF OBJECTIVES HIERARCHIES 



The hierarchies and SBO's were structured around the 
four F-4 missions: (1) Air to Surface, (2) Air to Air, 
(3) Reconnaissance, and (4) Escort. Each of the four missions 
was partitioned into the same ten mission segments: 



1. 


Premission Phase 


2. 


Brief Phase 


3. 


Prelaunch Phase 


4. 


Takeoff/Departure Phase 


5. 


Navigation Phase 


6. 


Tactical Phase 


7. 


Refueling Phase 


8. 


Approach/Landing Phase 


9. 


Post Mission Phase 


10. 


Debrief Phase 



It was from this basic structure of four missions by ten mission 
segments that the hierarchies and 580 were developed. 

The numbering of the SBO's indicates the hierarchical 
relationships among the SBO's which make up the hierarchy dia- 
grams. The first number specifies one of the four missions cited 
above while the second number indicate:^ one of the ten mission 
segments within the mission, e.g,, 1.1 is mission one (Air to 
Surface), mission segment one (Premission Phase), The subsequent 
digits in the SBO numbers are derived from the structures of the 
hierarchies and relate the prerequisite, lower-level component 
skills and knowledge to the higher level skills and knowledge. 

The hierarchies are numbered from top to bottom and within 
a branch at the same level from left to right. Each subordinate 
objective in a hierarchy has a reference number which includes 
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the complete reference number from its superordinate objective 
in the hierarchy and an added entry (Figure A-1). If multiple 
objectives are immediately subordinate to the same objective, 
the reference numbers for all of them will be identical in all 
but the last entry. The final entries for the subordinate 
objectives are sequentially assigned numbers beginning with 1. 

In Figure A-1 the first hierarchy for Mission Segment 1.1 
is shown. In order to conduct premission planning the pilot must 
"Gather Data for Mission" (1.1.1), "Select Pubs for Gathering 
Mission Data" (1.1.2), "Compute Air to Surface Mission Data" 
(1.1.3), "Record Data for Mission" (1.1.4), and "Evaluate Data 
on Mission Planning" (1.1.5). 

Three different types of blocks are shown in Figure A-1. 
The dashed rectangular block at the top of the page (1.0) in- 
dicates that the page is a continuation of another hierarchy 
page. The objective within a dashed block is that objective 
which is immediately superordinate to the objectives on the 
current hierarchy page. 

The first solid rectangular block following a dashed 
rectangular block (1.1 in Figure A-1) is always an objective 
which has been carried over from a previous page, i.e., there was 
insufficient room on a previous page to adequately analyze the 
objective, therefore, it was continued on a subsequent page. 
Other terminal solid rectangular blocks on a page are not analyzed 
further on a subsequent page. (1.1.4.1 ana 1.1.4.2 in Figure A-1). 

The "hex" box is used to indicate an objective which is 
further analyzed on a succeeding page (1.1.1, 1.1.2, 1.1.3, and 
1.1.5 in Figure A-1). The objective in a "hex" box is contained 
in the top solid rectangle on a subsequent hierarchy page: The 
superordinate objective to it is included in the dashed rectangle 
at the top of that page. 

Figure A-2 is a further analysis of the hex box 1.1.2 in 
Figure A-1. Note that the immediately suoerordinate objective 1.1 
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Figure A-1. Example Hierarchy: First Hierarchy in Preimssion 
Planninq Phase of Air to Surface Mission 
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Figure A-2. Example Hierarchy: Continuation Hierarchy 
from Hex Box 1.1.2 in Figure A-1 
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is contained the .dashed rectangle at the' top of the page 
and that 1.1.2 which was a hex box in Figure A-1 is the first 
solid rectangular box following the dashed rectangular box. 
In Figure A-2 all objectives are contained in solid rectangular 
boxes thus indicating that they will not be analyzed further on 
succeeding pages. 

One or wore SBO pages accompany each hierarchy. The SBO 
listing contains all information (i.e., numbei',' complete name, 
conditions, and standards) for all objectives in solid rectan- 
gular boxes in the accompanying hierarchy. For objectives in 
"hex" boxes, which wil'l be-analyzed further on-another page, 
"Only the number and objective name is present on the SBO page. 
'~ Figure A-3 is the accompanying SBO page for the hierarchy shown 
in Figure A-1; Figure A-4 accompanies the hierarchy in Figure 
A-2. 

If the same objective appears on more than one objective 
page it has alternate reference numbers. The duplication of 
an objective on different" hier-archy pages is indicated in two 
ways. First, such an objective contains a small solid triangle 
in the upper left corner of the reference number section of the 
box (Figure A-2). Secondly, all objectives with an alternate 
reference mark ar^ listed in an alternate numbers table. There 
is one such table at the end of the pilot hierarchies and one 
at the end of the RIO hierarchies. 

As the hierarchies were being developed clear patterns 
emerged within mission segments, across missions. In many cases 
the hierarchies and SBO's comprising a segment within one mission 
wore identical to those in the same mission segment within one 
or more other missions. For example, tne hierarchies in the 
Prelaunch segment were identical across all four missions for 
^nth the pilots and RIC's. Within both the pilot and RIO mission 
EKJCjiTients, 17 of the 40 mission segments were unique with the 
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GATHER DATA FOR 
MISSION 
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SELECT PUBLICATIONS 
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Figure A-3. AccoijTpfnyinq SBC's for the Hierarchy in Fiqure A-1 



lo: 



MISSION: Ai;^ TO SURFACE 

SEGMENT: COND.UCT PREMISSION PLATiING PHASE 



1,1.2 



1.1.2.1 



1.1.2.1.1 



] 1 ? 



ACTION • 

SELECT PUBLICATION'S 
FOR GATHERING MISSION 



CONDITIONS 



GiVE^I DATA NEEDED DURING PRE- 
^l:SS:ON PLANNING 



STANDARDS 



SELECT AIRCRAF^ DATA . G:,E:, DATA NEEDED DURING PRE- 
PUBLICATIONS : :^JSS;ON PLANNING 



SELECT TAKE OFF AND 
LANDING PUBLICATIONS 



GIVEN DATA NEEDED DURING PRE- 
N-IISSION PLANNING 



AS REQUIRED TACMAN 
-jR.^IATS AND BRIEFING 
G.IDES 

AS REC'.IRED Br TAC^IAN 
FORMATS AND BRIEFING 
GiIDES 



STA'E AIRCRAR DA'A . GIVEN A '^RI^TEN/ORAL EXAM . ^0 CRITERION 
PUBLICATIONS ' 



;S REQUIRED 3V TACMAN 
FORMATS AND BRIEFING 

L 4 L> ^ H<' 



1.1.2.2.1 ' STATE AIRCRAFT DATA 

■ PUBLICATIONS --^ 

1.1.2.3 : SELECT NAVIGATIONS 

; PUBLICATIONS 



1.1.2.3.1 ; STATE AIRCRAFT DATA 



1.1.2. 'i 



■ PUBLICATIONS 

t 
( 

: SELECT 'MiSSION 
; PUBLICATIONS 



1.1.2.4.1 : STA'E AIRCRAFT DATA 
; PUBLICATIONS 



GIVEN A WRITTEN/ORAL EXAM 



GIVEN DATA NEEDED DURING PRE- 
MISSION PLANNING 



GIVEN A WRITTEN/ORAL EXAM 



GIVEN DATA NEEDED DURING PRE- 
MISSION PLANNING 



GIVEN A WRITTEN/ORAL EXAM 



CRrERION 



AS REQUIRED BY TACMAN 
FORMATS AND BRIEFING 
GUIDES 



TO CRITER 



AS REQUIRED BV TACMAN 
FORMATS AND BRIEFING 



GUIDES 



'0 CRITERION 



FiGure n4. Accompanyir.q SBO's for the Hierarchy in Fiaure A-2, 
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remaining 23 segments being duplicates of the 17 unique segments. 

Due to the high degree of redundancy across mission segments, 
the pilot and RIO hierarchies contained only those which were 
unique. For both the pilot and RIO hierarchies Mission 1.0 (Air 
to Surface) is presented in its entirety, i.e.,<all ten mission 
segments are included. For the remaining three missions 2.0 
(Air to Air), 3.0 (Reconnaissance), and 4.0 (Escort) only those 
mission segments whose hierarchies differ from those in the 
corresponding mission segment in Mission 1,0 were presented. 
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APPENDIX 3 



MEDIA DEFINITIONS AND SAMPLE SELECTION DIAGPJ^.MS 



This appendix contains the media for each of the 
terminal blocks in the selection diagram, definitions 
of each of the candidate media, and examples of the 
selection process. 
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CANDIDATE MEDIA 



Each path in the diagram shown in Figure 8 leads to 
recommended niecia (l£:beled Ml through M4^-) for presenting 
the behavioral objective. These recosnmended media are shown 
in Figure B-1. In most cases primary and alternate media 
are recommended. However, in some cases one or more possible 
answers to a question are inappropriate for any objective 
that has entered a certain pathway. In these cases (such 
as Media ffl) the terminal blocks indicate that the combin- 
ation of answers to questions violated a principle of media 
selection. The remedial solution is to reviev; the path and 
correct the inconsistency. 

MEDIA DEFINITIONS. 

The instructional media contained in the model are: 

° Mediated Interactive Lecture 
■ ° Workbook 

° Slide Tape Presentation 

° Random Access Slide Presentation 

Videotape Presentation 
° Computer Assisted Instruction 
^ Trainer Exercise 
^ Aircraft Flight 
Each of the moclio is defined below. 

Mediated Interactive Lecture jMll ) - The major portion of 
the instructional material in an MIL is presented verbally by an 
instructor to a group of students. A MIL requires two hard copy 
products, student worksheets and visual aids. The student worksheet 
includes a set of lesson objectives, the generalities for each 
objective, necessary charts, tables, and figures, and a set of 
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Med T a #1 




Familiarization level content should not be 
at a recal 1 1 evel . 


taught 


Media wZ 




Familiarization level content should not be 
a rule-using level . 


taught 


Media ^3 




Paired associate level content should not be 
ax a ruie-using levei. 


taught 


Media 7t4 




First Choice: Workbook 




Media #D 




First Choice: Mediated interactive lecture 
oecono v^noice. oii oe— uape presen ua l i on 




Media #D 




First Choice: Videotape 

Warnirg: It may not be worth the expense 




Med 1 a # / 




It is probably a waste of time and resources 
this objective at a fami liarization level . 


to teach 


Media #8 




First Choice: Slide tape 

Second Choice: Mediated interactive lecture 


i 


Media #9 




First Choice: Videotape 

Second Choice: Mediated interactive lecture 




Media ^^10 




Why is an interactive presentation needed to 
discriminated recal 1 level behavior? 


teach 



Figure B-1 . The Meaning of Each of the 44 Terminal 
Blocks in the Media Sc'lection Network 
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Media //II 



You may want to combine this objective with the 
classification level objective dealing with this 
content. 

F i rst Choi ce: Workbook 

Second Choice: Mediated i r^tor-^ct i ve lecture 



Media HIZ 



First Choice: 
Second Choice; 



Vi deotape 

Mediated i nteracti ve lecture 



Media #13 



Why do you need an interactive simulation to teach 
a discriminated recal 1 1 evel behavior? 



Media #14 



First Choice: 



Second Choice: 



Media #15 



First Choice: 
Second Choice: 
Third Choice: 



Media #16 



First Choice: 
Second Choice: 
Third Choice: 



Media 



17 



First Choice: 
Second Choice: 
Third Choice: 



Media #18 



First Choice: 
Second Choice 
Third Choice: 



Media ^19 



First Choice: 
Second Choice 



A simulator or the 
and a worksheet. 
CAI simulation 



actual equipment 



Workbook 
Slide tape 

Mediated i nteracti ve lecture 



CAI (memory) 
Workbook 
Slide tape 



CAI (memory) 
Slide tape 
SI ide tape 



CAI (memory) 
Slide tape 
Workbook 



Vi deotape 

Mediated i nteracti ve 



lecture 



Figure B-1 



The Meaning of Each of the 44 Terminal 
Blocks in the Media Selection Network 
(continued) 
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Media ffZO 



Why do you need an interactive presontation to teach 
a discriminated recall level behavior? 



Media #21 



You may want to combine this objective with the work- 
book port";on of the rule-using level objective dealing 
with the content- 
First Choi::e: Workbook 

Second Choice: Mediated interactive lecture 
Media #22 

First Choice: CAI (incrnory) 
Second Choice : Wor^khook 
Third Choice: Slick* Lajje* 

M edia #23 

You may want to combine this with the workbook portion 
of the rule-using level objective dealing with this 
content- 
First Choice: Slide tape 
Second Choice: Workbook 

Media #24 

First Choice: Videotape 

Second Choice: Mediated interactive: locLure 
Media #25 

First Choice: Videotape 

Second Choice: Mediated interactive lecture (with VT ) 
Media ff 26 

First Choice: Siiiiu I a Lor 
Second Choice: CAI simulation 
Third Choice: Videotape 

Media #27 

First Choice: Videotape 

Second Choice: Mediated interactive lecture (with VT) 
NOTE: Be sure you have a separate objective to teach 

fche large memory component of this objective at 

the discriminated recall level. 



Figure B-1. The Meaning of Each of the 44 Terminal 
Blocks in the Media Selection Network 

(continued) 
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Media #28 



Fir<^t Choice: 
Second Choice: 
Third Choice: 



Media i29 



First Choice: 
Second Choice: 
Th>r<i Choice: 
Fourth Choice: 



Media #30 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 



Media #31 



First Choice: 
Second Choic?: 
Third Choice: 
Fourth Choice: 



Media #32 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 



Media #33 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 



Media #34 



First Choice: 
Second Choice: 
Thiy'd Choice: 
Fourth Choice: 
NOTE : 



Simulator 

CAI Simulation 

Vi deotape 



Workbook 
CAI 

Mediated' interactive lecture 
Random access slide-workbook 



CAI 

Random access slide-workbook 
Workbook 

Mediated i nteracti ve 1 ec ture 



Random access slide-workbook 
CAI 

Mediated i nteracti ve 1 ecture 
Workbook 



CAI 

Random access slide-workbook 
Mediated interactive lecture 
Workbook 



Workbook 
CAI 

Mediated interactive lecture 
Random access slide-workbook 



si ide-workbook 



CAI 

Random access 
Workbook 

Mediated interactive lecture 
Be stare you have a separate objective to teach 
the large memory component of this objective at 
the discriminated recall level. 



Figure B-1 



The Meaning of Each of the 44 Terminal 
Blocks in the Media Selection Network 
(continued) 
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Media f^35 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 
NOTE: Be sure 



Random access slide-workbook 
CAI 

Mediated interactive lecture 
Workbook 

you have a separate objective to teach 



the large memory component of this objective 
at the discriminated recall level. 



Media #36 



Fi rs t Choi ce : 
Second Choice: 
Third Choice: 
Fourth Choi ce : 
NOTE: Be sure 



CAI 

Random access slide-workbook 
Mediated i nteracti ve 1 ecture 
Workbook 

you have a separate objective to teach 
the large memory component of this objective 
at the discriminated recall level. 



Media #37 



First Choice: 
Second Choice; 
Third Choice: 
Fourth Choi ce : 



Media #38 



First Choice: 
Second Choice 
Third Choice: 
Fourth Choice: 



Workbook 
CAI 

Mediated interactive lecture 
Random access slide-workbook 



CAI 

Random access slide-workbook 
Workbook 

Mediated interactive lecture 



Media #39 



First Choice: 
Second Choi ce: 
Third Choice: 
Fourth Choice: 
NOTE: 



Workbook 
CAI 

Mediated interacti ve 1 ecture 
Random access slide-workbook 
Be sure you have a separate objective to teach 
the large memory component of this objective 
at the discriminated recall level . 



Media #40 



First Choice: 
Second Choice : 
Thi rd Choi ce ■ 
Fourth Choi ce : 
NOTE : 



CAI 



Random access slide-workbook 
Workbook 

Mediated i nteracti ve 1 ecture 
Be sure you have a separate objective to teach 
the large memory component of this objective 
at the discriminated recall level- 



Figure B-1. The Meaning of Each of the 44 Terminal 
Blocks in tjie Media Selection Network 
(continued) 
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Media #41 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 



Media ffAZ 



First Choice: 
Second Cho i ce : 
Third Choice: 
Fourth Choice: 



Random access slide-workbook 
CAI 

Med i a ted interactive 1 ec t u re 
Workbook 



CAI 

Random access slide-workbook 
Mediated i nteracti ve 1 ecture 
Workbook 



Media 



First Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 
NOTE: 



Random access slide-workbook 
CAI 

Mediated interactive lecture 
Workbook 

Be sure you have a separate objective to teach 
the larje memory component of this objective 
at the discriminated recall level- 



Media #44 



Firs'*" Choice: 
Second Choice: 
Third Choice: 
Fourth Choice: 
NOTE: Be sure 



CAI 

Random access slide-workbook 
Mediated interactive lecture 
Workbook 

you have a separate objective to teach 



the 1 arge memory component of thi s object i ve 
at the discriminated recall level. 



Figure B- I . The Meaning of Each of the 44 Terminal 
Blocks in the Media Selection Network 

(continued) 
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practice item response sheets for each objective. Types of visual 
aids used in a MIL may include overhead transparencies, slides, 
or videotapes, depending on the display requirements of the lesson. 
These aids are used, where appropriate > to present supporting 
information, sets of examples, and sets of practice items. For a 
lecture to be adaptive to student needs it must be interactive. 
An interactive lecture provides the instructor with an outline 
which requires him to ask the students questions and a. ter his 
presents ti on based on student responses , 

Workbook (WB ) - The defining characteristic of a workbook is 
that all instructional components are presented in printed form. 
The workbook begins with a lesson map, a lesson introduction, and 
a set of instructions explaining how to proceed. Each segment 
(corresponding to one objective) begins by presenting the objective, 
the generality, and the supporting explanation or help. The 
practice items are designed to require the student to behave at the 
level designated in zhe objective. The practice set is accompanied 
by a feedback and help section. 

Slide -Tape Presentctlon (ST ) - A slide-tape presentation shall 
contain three separate components: (1) a set of slides or photographic 
frames, (2) audio sections, and (3) a student worksheet. The 
student worksheet first presents a lesson map, and a set of 
instructions about where to get the slides and the audio tape-, 
where ana how to set them up, and how best to use them. For each 
Segment , the worksheet presents the object i ve , the general i ty , 
necessary charts and tables, and a response form for the practice 
items. The audio tape shall begin with an introduction. For 
each objective, the generality is restated and additional supporting 
information is presented. The slides shall c.:)mplement the audio 
tape. A set of practice iterr^s is presented next. The student 
is told to stop the tape after each question, and write his 
answer on his worksheet. When the tape is started again, it gives 
feedback and help for that item. 
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Rando m-Access Sli de Present ation (RAS^) - A random-access 
slide presentation shall contain two components: a set of slides 
presented via a random-access slide projector, and a student 
worksheet. This medium allows for visual presentation of large 
numbers of examples and practice i terns' wi thou t sacrificing the 
freedom of movement (i.e., learner control) lost in t'le linear 
ST or VT presentation. The worksheet in the RAS shall present 
the lesson map, a lesson introduction, and instructions for 
obtaining the slide set, setting up, and viewing, and how best to 
use the lesson materials. For each segment, it presents the 
objective, the generality and sonic supporting explanation. It 
then presents tables of numbered example items and practice items. 
Each table presents a listing of the slide numbers of example, 
and heip, or practice and feedback items. The student can use 
these tables to check those examples he has viewed, or the practice 
items he has worked. The practice table also provides space for 
the student's response. The random.-access slides are used to 
present an expanded version of the generality, the set of examples 
with helps, practice items, and feedback. 

Videotape fVesori La Li ori(_VT) - A videotape (:^res enta ti on shall 
contain two components; a videotape cassette and a student worksheet. 
The use of these two components shall be parallel to the use of 
corresponding components of the slide-tape presentation. The major 
difference between these two media is the type of display capability. 
The instructional strategy used is similar in both types of media. 

Computer Ass i s ted I nstruction (CAI ) - In this medium a 
computer monitors student performance through frequent tests and 
optimizes trainee learning by branching to the appropriate level 
of detail. When trainees elicit a strong understanding of the 
content the computer omits further instruction. When the trainee 
fails to master a subject the computer branches to more detailed 
instruction, much as ci human instructor does. Due to the high 



:i8 



EKLC 




NAVTRAEQUIPCEN 77-C-008i-l 



cost of computer assisted instruction, Allen Corporation wr.s 
instructed to omit CAI from consideration on this training 
program. 

Trainer Exercise - A trainer exercise shall be used to allow 
the student to practice in a simulated real world environment. 
The type of trainer used depends on whether cockpit riiOtion i 
required, and upon the degree of fidelity to the real world 
required. A trainer exercise shall also include three other com- 
ponents: (1) a student worksheet, (2) a student evaluation sheet, 
and (3) a trainer feedback sheet. The student worksheet presents 
the student with the set of lesson objectives, a lesson intro- 
duction, and a set of instructions describing what he should do 
to be prepared for the trainer. The worksheet then presents any 
information the student will need for mission planning purposes, 
and spells out exactly what planning he will need to have prepared. 
The worksheet then outlines in detail the procedures or actions 
the student should perform. For procedures not previously practiced, 
the worksheet presents a list of steps to perform. Previously 
mastered procedures are referred to by name without the accompanying 
list of steps. In all cases, the worksheet outlines verbally or 
pictorialTy the correct result of the action. The student evaluation 
sheet is used by the instructor to check the student's performance. 
It describes evaluation instructions, outlines necessary instructor- 
student interactions, and gives a checklist of all points that 
should be evaluated- Based on the student's performance, the 
instructor fills out a trainer feedback sheet- This sheet is 
formatted so the instructor can check areas where the student 
needs more practice. Each area references course lessons which 
deal with the area- 

Aircraft Flight (AC ) - The instructional materials for flight 
shall be similar to those for a trainer exercise. These materials 
include a student worksheet, a student evaluation form, and a flight 
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feedback sheetr^The student worksheet outi -M^ es t he lesson objectives, 
gives the student necessary data/-d£id instructionsf^Tsmiss ion 
iJlanninc], .and describes the general sequence of eventSs^at should 
occur during the flight. The student evaluation sheet and the 
flight feedback sheet are the same format as those in the 

trainer exercise. The student evaluation sheet shall be printed 
on knee-cards so it can be carried on the flight. 

SELECTION PROCESS EXAMPLES 

As discussed in Section III, media for presenting each objective 
were selected by asking a series^ of five questions. Answers to 
the questions traced. a path through the media selection diagram. 
Figure B-2 contdins a copy of the media selection diagram for 
pilot behavioral objective 1.3.4.1.1 (State Procedures for Taxi 
On Field). The dark lines indicate the questions asked about the 
objective and the pathways c-hosen based on the answers. 

The algorithm starts at the triangle in the center of the 
page and proceeds to Question 1 (What is the level of behavior 
expected of the student in this segment?). The answer is No. 2 
(discriminated recall) because the student is required to recall 
and state the procedures. 

The algorithm then proceeds to Question 2 (What level of 
content is being taught in this segment?). The answer is No. 3 
(concept) because the student must state the concept or pro- 
cedures but not manipulate it as in No-. 4. 

Next is Question 5 (Is the memorization component of this 
objective large or small?). The answer is No. 2 (large) because 
there is no checklist for taxi procedures and much information 
must be^remembered to correctly perform taxi operations. 

The final question is No. 4 (What is the minimum display 
requirement?). The answer is No. 2 (verbal and/or symbolic and/or 
static complex pictorial) because the long list of procedures 
will require complex static pictorial presentation. 
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First Choice: Slide tape 
Second Choice: '/lorkbook 

1.3'.4.1.1 STATE PROCEDURES FOR 
Trl OfJ FIELD 



Figure B-2. Media Selection Map for 
- Objective 1.3.4.1.1 
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The algorithm has led to Media 18. As shown in Figure B-1, 
tlie Ti rs t ~choi ce ~^for TTedia" 1 8 " i s—CAtIt Due to cost cans i dera t i ons 
this option was omitted. Consequently » the recommended media for 
presentation of the objective (State Procedures for Taxi On Field) 
are slide tape (first choice) and workbook ( second .choi ce ) . 

Figure B-3 contains a copy of the media selection map for 
pilot behavioral objective 1.3.4.1 (Perform Taxi On Field). The 
first question concerns the level of behavior required of the 
student. The answer is No, 3 (rule using) because the student 
must apply the rule in actual practice. 

Question No. 2 asks the level of content to be taught. The 
answer is No. 4 (rule) because the student must manipulate the 
concept. 

The last question (No. 4) asks what are the minimum display 
requirements. The answer is No. 4 (interactive) because the 
display system must sense student in[>uts and feedback actual 
aircraft responses. 

The algorithm has selected Media 14. ' e first choice for 
Media 14 is a simulator or the actual equip er t: and a worksheet. 
The second choice is CAI and was eliminated. 
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^ APPENDIX C 

ANALYSIS EXAMPLE 

This appendix contains an example which traces objectives 
"Trom the task listing, through hierarchy and SBO development 
and media selection, to the syllabus. 
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ANALYSIS EXAMPLE 

In the sequence of Figure C-1 through C-ii on the following 
pages, three objectives froru the task listing are traced through 
three analys is steps . First they are broken down into their 
prerequisite skills and knowledges in an objectives hierarchy. 
Next, media are selected for one of these objectives and tv;o of 
the prerequisite objectives, using media selection diagrams. 
Final ly* the objectives from the hierarchy are grouped into 
syllabus lessons*. 

The task listing page shown in Figure C-1 was the start 
point for the analysis. Objectives marked with asterisks (*) 
are those which are analyzed in the subsequent hierarchy. 
Figure C-2 is the hierarchy containing the superordinate objectives, 
the objectives af interest from the task listing, and the sub-- 
ordinate objectives identified during the hierarchy analysis. 
Ncte that objective 2.6.6.4 (Determine the Optimum Lesson to Use) 
is at the same level in the hierarchy as the three objectives from 
the rask listing. Although the inclusion of this objective 
resulted from analysis of the other three objectives at the same 
level, it was placed as a coordinate objective as per the 
specification guidelines for hierarchy development. Figure C-3 
is the SBO pages for the hierarchy in Figure C-2. 

The next step was to select instructional media for the 
objectives. Figures C-4, and C-6- are media selection 

diagrams for three of the objectivec in Figure C-2. 

Following media selection, the objectives were grouped to 
form lessons. Two lessons, one ground school and one flight, 
were formed from the objectives in Figure C-2. The syllabus 
sheets for these lessons are shown in Figure C-7 and C-8. 
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F-4 AIRCREW TASK LISTING VALIDATION 

FiLO T 



TASK CONDITIONS STANDARDS 



2.6.5.1.3 

Acquire TGTS using PLM 


Given tactical situation 


lAW TACMAN 


2.6.5.1.4 

Acquire TGTS using 


Gi van tactical si tuation 


lAW TACMAN 


2.6.5.1.5 

Acquire TGTS using 
radar 

(rear seat acquisition) 


Given tactical situation 


I AW TACMAN 


9 A 

Del i ver weapon 






2.6.6.1 ^ 

Del i ver Aim 7 


Gi ven tactical si tuati on 


I AW TACMAN 


2,6. 6. 2 ^ 
Deliver Aim 9 


Gi ven tacti cal si tuati on 


I AW TACMAN 


2.6.6.3 ^ 

Deliver 20mm gun fire 


Gi ven tactical si tuati on 


lAW TACMAN 



Figure C-1 . Sample Task Listing Page. Entries 

-Marked with an Asterisk (*). were the 
Superordinate Objectives for Subsequent 
Analysi s 
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coNoia tactical} 

rtl',SE i.r AN . 
AIR TO AIR I 
MISSION I 



rETFRMINE 
VIuN AIRCRAFT 
IS WITHIN 
LAl'NCIl 

E:a'ELo r£ 

2.6.6.1.1 



STATE 
FACTORS 
AKFlCTINC 
DECISION 

L6.e.:.i.i 



DELIVER 



:.6.6 




:?,OCE0'JRES 



2.6.6.1.2 



r>ETE?J':lNE 
l-.'HEM AIRCRAFT 
IS WITHIN 
LAUNCH 
ENVELOPE 



2.6.6.2.1 



STATF 

PROCEDLT.FS 



2.6.6.2.2 



oi:termine 

WiEN AIRCRAFT 
IS WITHIN 
DELIVERi 
EOTELOPE 



2.6.6.3.1 



STATE 
FROCiDLRES 



2.6.6.3.2 



STATE 
FACTORS 
AFFECTINC 
DECISION 

2.6.6.2.1.1 



STATE 
FACTORS 
AFFEiTIN'n 
PrXISION 

2.6.6.3.1.1 



dl:ef^;i:;e 

THE UPr:!fl'>! 
'.sTAPO;.' TO 
USE 



2.6.6.i 



STATE 
r-\CTQRS 
AFFECTK.T 
DECISION 



2.6.6.^<.l 



Z 
> 
< 
H 
73 
> 

m 
o 
c 

1-4 

■0 

Ci 

m 
z 

>j 

I 

o 

o 
o 

CO 



Figure C-2. Sa^r^ple Hierarchy Page. Entries marked with an asterisk (*) are superordinate 
objectives from the task listing. Other objectives at or below the level of 
the marked objectives are prerequisite skills and knowledges. 
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MlSSlOii: AIR TO AIR 

SEGHENT: COiUCT TACTICAL PHASE 



SBO? 



2.6.5 



2.6.6.1 



2.6.6.1.1 



2.6.6.1.1.1 



2.6.6.1.2 



DELIVER WEAPOfi 



DELIVER AIRBORi I^^- 
TERCEPT MISSILE 7 

DETERMINE AIR- 
CRAFT IS mm 

mm imm 

STATE FACTORS 
AFFECTir^G DECISION' 

STATE PROCEDURES 



2.6.6.2 j DELIVER AIRBORNE l}r 
TERCEPT MISSILE 9 



2.6.5,2.1 



2.6.6.2.1.1 



i 2.6.6.2.2 



DETERMir^E WHEN AIR- 
CRAFT IS WITHU^ 
LAUNCH ENVELOPE 

STATE FACTORS 
AFFECTING DECISION 

STATE PROCEDURES , 



CONDITIONS 



STANDARDS 



GIVEN TACTICAL SITUATIOf^ A^D TO WITHIN LETHAL 



ADVERSARY AIRCRAFT 

GIVEN TACTICAL SITUATION AND 
ADVERSARY AIRCRAFT ' 

GIVEN TACTICAL SITUATION AND 
ADVERSARY AIRCRAFT 



GIVEN EKAM 



WEAPON 

TO WITHIN LETHAL RADIUS OF 
WEAPON 

'v 

IN ACCORDAfiCE WITH TACMAN 



GIVEN EKAM 



TO CRITERION 



TO CRITERION 



GIVEN TACTICAL SITUATION AND ; TO WITHIN LETHAL RADIUS OF 
ADVERSARY AIRCRAFT 



GIVEN TACTICAL SITUATION AND ! IN ACCORDAfJCE WITH TACHAN 
ADVERSARY AIRCRAFT ; 



GIVEN U 



GIVEN U 



TO CRITER 



TO CRITERION 
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Figure C-3. SBO Page for the Hierarchy in Figure C-2 



MlSSIOf^: AIR TO AIR 

WB]: mmi lACIICAL PRASE 



SBO-: 



ACT! Of; 



CCf^DITIOf^$ 



i 2.5.6.3 i DELIVER 20 HILLIMEIER 1 SIVE!^' TACTICAL SITOA'IOfJ AfID ' iC '^:^m LETHAL RADIUS 



i iGOUIRE 

I 

: 2.6.6.3.: 'oETEf^lIf^E '.'HE 

: CRAFT IS '.'iTHIfl 
I 1 LAUfiCH E.V.ELC^E 

I ! 

I 2.6.6.3.1.1! STATE FACTORS 



I 2.6.6.3.2 

I 

' 2.6.6.4 



i 2.6.6.4.1 



STATE PROCEDURES 

OETERHINE THE 
OPTIMUM WEAPON 
TO USE 

STATE FACTORS 



ADVERSARY AIRCRAFT 



''M TACT 



OP ,^'EAFOf^' 

lu ACCGRDAflCE ^ITH TACi^l 



' ijVERSAPy AIRCRAFT 



I 



GIVE*^ E^AM 



GiVEf^ TACTICAL SI 
^VERSARy AIRCRAF 



GiVE^i E,\AM 



TO CRITERIOfi 



TO CRITERIO,^ 



TO CR 



^igyre C-3. SBO Page for the Hierarchv in Fiqure C-2 [copti^iiedl 
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Media =14 



First Choice: A si^Iator or 
the actual equipirient and a As \J r 
worksheet \— / 

t) 

2.6.6.1 DELIVER AIRBORNE ^ 
LmEPT MISSILE 1 



J. 03 



Q3 



r I 



6t 



Q3 iL, 
T 



z 
> 
< 

H 
> 

m 
o 
c 

M 

"0 

n 
m 

2 

SI 
SI 

I 

O 

I 

O 

o 

OS 



0 



p 6© 



0 
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Figure C4. Hedia Selection Diagram for One of the 
Superordinate Objectives in Figure C-2 




Nedia =8 

First Choice: Slide tape .;r~\ I 
Second Choice: Mediated inter- 'Y ^5 \J H 
active lecture ^ — ' 

2.6.6.1.1.1 STATE FACTORS AFFECTING 

DECISION (LAUNCH ENVELOPE) C 



Figure C-5. Media Selection Diagrani for One of the 
Subordinate Objectives in Figure C-2 




Figure C-6. Media Selection Diagram for One of the 
Subordinate Objectives in Figure C-2. 



Course, Unit, Lesson: P. 17.1 
Title: MISSILE SHOOT 



Time: .5 
Medium: ST 
A/C Model: J/N 
Organization: M/N 





1 

i ^ CTMCMT T FTI C 


1 con MTi 

I , - 


1 SEL. 
i MEDIA . 


1 

■ 1 


jidie TdCiOrs nTTecnng aecision pauncn enveiopej 


i 0 P. Q 1 } ^ 


i 

1 1 


0 

c 


C^i^Q n V* A 1* riH ■ 1 c ( ATM 7 ^^rl^^^U\ 

jldLc prOCcOureS ^Min-/ launcnj 


! 9 ^; ^ 1 9 




J 


iicLc Tdciors dTTecLmy QGCision ^ iduncn ertveiopej 


i 

1 9 ^ 9 1 1 
\ L. 0,0. L.Li 

1 


j 




jLdLc prULcuLireb [hin')jj 


0 fs 0 0 

f 
1 


j 

1 


c 


jLaic TcCtors aTiecnny uecision ^launcn envelope/ 


1 0 f^ \ ^ 
j 1. D.D.J. 1*1 




6 


State procedures (for 20MM) 


I 2,6.6.3.2 




7 


Determine optimum weapon to use 


i 

1 2.5,6.4.1 

i 


RAS 

t 


8 


State factors affecting decision (optimum weapon) 


2.6.6.4.1 


1 
i 


9 

1 


State means of evaluating battle damage assessm.ent 


2.6.7.1 


RAS 

[ 



Figure C-7. Sample Syllabus Sheet. Segments 1 through 8 
were Grouped from the Hierarchy in Figure C-2 




Course, Unit, Lesson;. P.17.3 Time: 1.2 

Medium: F-4 

Title: MISSILE SHOOT A/C Model: J/N 

Organization: M/N 



SEG = 

V w u 


' ' ■ ' ■ ■ - - ■ - — r 

SEGMENT TITLE { • 


SBO NO 


<£L ' 
MEDIA 1 


1 


Deliver weapon ! 


2.6.6 . 


i 

1 

1 


2 


1 

Deliver AIM 7 j 


2.6.6.1 


i 
1 
1 


'; 3 


1 

Determine when aircraft is within launch envelope ' 


2.6.6.1.1 


1 




Deliver AIM 9 | 

i 


2.6.6.2 


i 
1 

1 




Determine when aircraft is within launch envelope | 

i 


2.6.6.2.1 


! 


6 


Deliver 20MM 


2.6.6.3 


i 




j 

Determine when aircraft is within launch envelope | 


2.6.6.3.1 


i 
1 


i 
1 


Evaluate results of weapon delivery \ 

1 
1 

1 
1 

t 

1 


2.6.6.7 


{ 
t 

I 
\ 

i 



Figure C-8. Sample Syllabus Sheet. Segments 1 through 7 
were Grouped from the Hierarchy in Figure C-2 
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APPENDIX D 



TRAINING SUPPORT REQUIREMENTS ANALYSIS 



This appendix coritains extracts from the Training 
Support Requirements Report (CDRL Item EOOll). Sections 
presented pertain to cost estimation procedures and the 
figures used in the calculations. 



NAVTRAEQUIPCEN 77-C-008]-! 
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The foUowing sections of this appendix contain the basic 
information which was used to estimate costs for development, 
implementation, revision and maintenance of the F-4 pilot and 
RIO courses. Since the design phase was under contract, when 
the TSRA was prepared, no estimate for this phase was calculated 
Of particular relevance are the development costs for one lesson 
in each of the seven media selected for use in the F-4 training 
courses. As discussed in Section III of this report, the 
number of lessons per medium has changed as a result of lesson 
specification development, which began after the original TSRA 
was produced. Total estimated costs, therefore, have changed, 
but not the costs per lesson in each medium. 

DEVELOPMENT REQUIREMENTS 

The development phase consists of the authoring and pro- 
duction efforts. Base rate values for these efforts we're 
developed in the format specified in UDI-H-Q0040. Table D-1, 
Personnel Hours and Materials to Develop One Lesson .of In- 
struction, contains the basic data required. Data in Table D-1 
were drawn from three civilian production companies and from 
two civil service (Navy) organizations. 

In Table D-1 the 5ME has b .en considered to be the author. 
The Instructional Psychologist has been considered as a part- 
time program manager with the Instructional Technologist 
providing most of the direct support and assistance in the 
authoring and development efforts. The production manager pro- 
vides the liaison between the graphics, camera, and typing 
personnel. In addition, he will provide the scheduling of 
input and output for the storyboard and illustration requirements 
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T'tBLE D-1 . BASIC DATA USED TO CALCULATE 
nEVELOPMFNT COSTS 













SMC 










Writer/ 




























1 




u 

e 

c 

T 
U 
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t 






5 


. 5 








j 1 










! 


8 


. 5 


i 2 


.5 j 2 


1 , 5 




1 


1 

! .5 




^■ 


2 




1 

! 


1 








[ 30 


1 - 


•OOK 






30 




i , 


1 


5 


1 

i 5 






1-2 


•OOK 
























SUIOC 






3 


1 


2 


6 








so 


i 

5 








c 






2 


6 


1 - 5 










o 


I 




1 


2 


2 




10 


2.5 


*cc 






" 6 


2 










j 


160 


iO 










■p 


3 


1 


4 


5 




20 

t 


5 


< 








1 


4 


2 




3 


1 

1 1 


40 




> 








.5 


2 


.5 




1 1 




i 

5 ! 


1 








c. 


. 2 


.3 


. 5 


^ \ 


2.5 




10 


2-5 






1 

1 


1 


.2 


. 3 


-5 ' 


,2 
















1 




. 3 




.2 






( 

. ! 


1 








2 


,2 


-3 


.5 


.2 


2.5 




10 j 


2.5 






i 
1 




.2 


! 


■^ i 


T j 




1 

f 








1 

1 
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.2 


t 
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.5 


- ! 


I j 
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1 


5 





ERIC 



NAVTRAEQUIPCEN 77-C-0081-1 



TABLE D-1 . BASIC DATA USED TO CALCULATE 
DEVELOPMENT COSTS (CONTINUED) 



















1 

j o.... 1 


1 


















< 












1 
1 

! 2 


S 23-. GO 


























1 37.00 




i - 
















2.5 








$ 20.00 


J . - 










S I - GO 






1 n 






92.00 


100.00 




i 












i i 
































243.00 












. 15 






1 










S 9.45 


t 
I 
1 




S 2.50 




i — 






I 






46.00 


45.00 




I 

1 
1 












so 

1 


u- . . 






4a6.00 




1 








1 Zi 




! H 

i ' 






9Z.0O 


90.00 




























S 4a. 00 






$ 25.00 










2 


23. or) 


23.00 




















4 j 46.00 


45.00 




J 


















23-00 








1 








1 

1 


r> J 
1 




2 


23.00 


23.00 


i 1 
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\ 

4 i 

• 




4 


■i6. 00 


45. 00 










j 










1 . S 


23.00 














1 




i 
1 




2.0 


23.00 j 


2 * , 00 j 






1 

I 
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The personnel hours shown in Table D-1 were converted to 
estimated "loaded" salaries in Tables D-2 through D-7. The 
Approximate Hourly Rate contains the estimated additions for 
overhead, G&A (General and Administrative) and profit fees. 
Tables D-2 through D-7 show the estimated costs, by media, 
for production of the lessons in thf? r-4 ^jilcju and RIO sj/llal>i. 

Mater'als and services are indicated In Tables D-2 through 
D-7 using dollar amounts. The Development Total represents the 
cost for one reproducible master of finished lesson materials 
for the given medium. Duplicatior costs were not estimated 
since duplication will be carried out oy the government-- 

A survey of available training assets (equipment) conducted 
during Media Selection and shown in Table D-8, indicates that 
there is adequate equipment for disp*: ay of Videotape and adequate 
simulator/trainer and aircraft availabilities. However, there is 
a requirement for slide/ tape machines.' Additionally, it has been 
determined that MCAS , El Toro and NAS , North Island have adequate 
Videotape capabilities as well as reproduction capabilities tnat 
could be utilized during the development effort. Further research 
determined that Code N-51, Navy Education and Training Support 
Center, Pacific has facilities for duplication of 35mm „sl ides. 
Tasking and scheduling would be, required through Code CNET. 
Duplication of paper materials could be accomplished through 
the Navy Publication and Printing Service and tasked through 
OPTAR. No specific in-service organization appeared to have a 
total aevel opment_ crew that could be made available for full F-4 
training development. It is recommended that the production of 
all original materials be contracted and that duplication of 
the originals be done by the government . 

The equipment required for development are those items 
associated with a full graphics and photography capability and 
include: cameras, lay-up tables, back-light table, photo^copiers , 
slide/tape mach i ne , audi o pulsing and recording machines, etc. 
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TABLE D-2. COSTS FOR'PRODUCTION OF . 

MEDIATED INTERACTIVE LECTli'RES 



\ 





1 


, 1 HOURS 


■V 


•PfRSONNEL 


' HOURLY 


' : PER 1 F^STiN 




i RATE 


1 : TABLE 3 • 


$ PER LESSON 


SME/AUTHOR 


) 


(15) 


i 


LP. /PROGRAM MGR. 


: $30.00 


1 i 1 


1 

S 30 DO 1 


I. T. /DESIGNER 


- 22.00 


: : 3.5 


77 nn i 


PROO.UCTION MGR. 


; 30.00 


; : 2 


60.00 


EDITOR/PRCOF 


I 18.00 ' 


; ' 3 


54.00 


TYPTST/COMP. 


10.00 


3 


30 00 


NARRATOR 


i- 70.00 i 






GRAPHICS SPEC. 


; 22.00 


: 32.5 


715.00 


PASTE-UP ARTIST 


12.00 


! ^ 2.5 


30.00 


Dunin/'D fl Due 3 • 
rnJI UbKArHtK 


; 19.00 


i : 2.5 


47,50 j 


T,V. DIRECTOR 


25.00 


i : 


1 


PRINTER 


12.00 

! 
i 
1 


I 

- i 


50.00 


PERSONNEL TOTAL 




i 55.0 


$1103.50 ! 


MATERIALS/SERVICES 








60.00 

i 


1 

1 

DEVELOP. TOTAL j 




1 






$1163.50 


1 




-H-H-r^ ^ L 


1 

1 
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TABLE D-3. COSTS FOR PRODUCTION' OF WORKBOOKS 



PERSONNEL 



MATERIALS/SERVICES 



DEVELOP. TOTAL 
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HOWLY 
•RATE 



HOURS 
'PER LESSON 
(TABLE .3) 
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$ PER LESSON 



SME/AUTH^'^R 


/ j 


(30j 




I. P. /PROGRAM MGR 




1 ■ 


$ 30.00 


I.T /DESIGNFR 






• ■ B'B.OO 




1 j 

30.00 -! ; 


1 


30.00 


EDITOR/PROOF ' 


18. ao ; 




90.00 


TlfPIST/COMP. 


10. "00 j : 


s 1 

i 


50.00 


NARRATOR 


70.00 ; 


I 




GRAPHICS SPEC. 


22.00 : 

1 1 


20 ! 


440.00 


PASTE-UP ARTIST 


12.00 1 ; 


j 

1.2 1 


14.40 


PHOTOGRAPHER 


19.00 ; 


10 i 

1 


■ 190.0.0 


T.V.,DIRECTGR 


25.00 . : i 


1 
t 

j 




PRINTER 


12.00 ; 

I 1 

1 i 

i 

■ t- 


8 I 

! 

1 


96.00 


PERSONNEL TOTAL 


1 ! 


55.2 i 

1 

1 


. $1028.40 
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$1220.40 
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TABLE D-4. COSTS FOR PRODUCTION OF 
SLIDE-TAPES 



PERSONNEL 


i 

HOURL'r 
RATE 


1 HOURS 
i PER LESSON 
; (TABLE 3) 


1 

i 

i $ PER LESSON 


SNE/AUTHOR 




H 

1 

(39.5) 


1 
1 


I, P. /PROGRAM MGR; 


$30.00 1 


1 4 


$ 120.00 


IJ, /DESIGNER 


1 j 

22.00 j 


14 


308.00 1 


PRODUCTION MGR. 


30.00 




i 

330.00 


EDITOR/PROOF 


18.00 


i 4 

i 


72.00 


TYPIST/COHP. 


10.00 


8 


80.00 


NARRATOR 


70.00 


r 

1 . : 


105.00 


GRAPHICS SPEC. 


22. OC 




1980.00 


PASTE-UP ARTIST 


12. OC 


'.5 


50.00 


PHOTOGRAPHER 


19.00 




836.00 


T.V. DIRECTOR 


25.00 






PRINTER 


12.00 


! 4 . 1 


4R.nn 



PERSONNEL TOTAL _ 
MATERIA.S/SERVICES 



184 



$3969.00 



DEVELOP. TOTAL 



J 



ERIC 



4 n 



$4303.00 



\ 



TABLE 0-5. N:0STS FOR PRODUCTION OF VIDEOTAPE 



PERSONNEL 


\ 

\ 

HOURLY x 
RATE \ 




HOURS 
PlR LESSON 
(TABLE 3) 


$ PER'LESSON 


SME/AUTHOR 




V 

■ 


, (32) 




I. P, /PROGRAM MGR. 


, $30.00 




1 . 5 


$ 4b. 00 


I.T. /DESIGNER 


22. 00' 




'^, 6 


132.00 


PRODUCTION MGR. 


30.00 


i '2 5 
1 \ 


75.00 

• 


EDITOR/PROOF 


18.00 


: 3, 

! 


54.00 


TYPIST/COMP. 


10.00 


: 


40.00 


NARRATOR 


70.00 j 


2 


140. 00 


GRAPHICS SPEC. 


22.00 




, i 


Qon on 


PASTE-UP ARTIST 


12.00 




1 \ ; 


k.OO 


PHOTOGRAPHER 


19.00 


: 

1 


4 ] 

t 


76.00 


T.V. DIRECTOR 


25.00 i 


1 2 1 


50.00 


PRINTER 

i 


12.00 


' 2 ' 

I i 

t 

1 

.! 


'\ 24.00 


' 1 

1 

PERSONNEL TOTAL 


! 




73 

i 


$1638.00 


MATERIALS/SERVICES 


» 

; 1 


1 

1 

j 

1 


92.00 


DEVELOP. TOTAL \ 

1 1 


i ' 


$1730.00 ■ 
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TABLE D-6. COSTS FOR PRODUCTION OF- 

TRAINER/SIMULATOR MATERIALS 



PERSONNEL 


1 

HOURLY 
RATE 

1 


I 


i HOURS 

PER LESSON 
I (TABLE 3) 


1 


SME/AUTHOR 




! ! w 

1 1 


i 
i 
i 


I. P. /PROGRAM MGR, 


i $30.00 ' 

1 


! .6 

1 


; $ 18.00 


I. T. /DESIGNER 


22.00 


1 


i .9 

1 


19.80 

1 


PRODUCTION MGR 


i 30.00 


i 1.5 


45.00 


EDITOR/PROOF 


18.00 i 


1 

.6 

] 


i 10.80 


TYPIST/CCMP 


10.00 


! 4.5 


45.00 


NARRATOR 


70.00 


( 

1 


1 


GRAS'HICS SPEC 


22.00 i 


1 ^ 


66.00 


PASTE-UP ARTIST 


12.00 




1 .5 


■ 6.00 


PHOTOGRAPHER 


19.00 i 


i ' 


38.00 


1 

T.V. DIRECTOR 


25.00 


1 

1 




PRINTER 


12.00 ! 

1 

1 

J 


1 7.5 . 

i 


90.00 


PERSONNEL TOTAL 
MATERIALS/SERVICES 


1 
1 

i 




21.1 ! 5338.60 

1 

(Guides, Check- 115.00 
lists a Worksheets) 


! 

DEVELOP. TOTAL 

i 




i 


$453.50 

i 



TABLE D-7. COSTS FOR PRODUCTION OF CHECKLISTS 
AND WORKSHEETS FOR FLIGHTS 



PERSONNEL 


HOURLY 
RATE 




HOURS 
PER LESSCN 
(TABLE 3) 


$ PER' LESSON 


SME/AUTHOR 






(4) 


• 


i.9.i?mm. MGR. 


$30.00 




.6 

1 


$ 18.00 


I. T. /DESIGNER 


22.00 




1 .9 


19.80 


PRODUCTION MGR. 


30.00 




1.5 


45.00 


EDITOR/PROOF 


18.00 




.6 


10.80 


TYPIST/COMP. 


10.00 




4.5 


45.00 


NARRATOR 


70.00 








GRAPHICS SPEC. 


22.00 




3 


66.00 


PASTE-UP ARTIST 


12.00 




.5 


6.00 


PHOTOGRAPHER 


19.00 


i 


2 


38.00 


T V DIRF TOR * 


25 00 


1 




PRINTER 


12.00 i 


i 

i 
1 


7.5 I 


?0.00 


PERSONNEL TOTAL 
MATERIALS/SERVICES 


j 
i 


1 

1 

! 


21.1 

1 

1 


$3?S.60 
115.00 


DEVELOP. TOTAL 


1 

,„ . . . ... 1 


! 




$453.60 
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TABLE D-8 . SURVEY OF AVAILABLE TRAINING ASSETS - June 1977 



AIR 



J 



STATIONS ,2F88'2F55 :i5ai2C17 



N 

2C30 



TRAINERS m MEDIA EQUiPKENT 



SLIDE MOVIE 



FROJ. 



FILM 
TAPE 'strip 



VIDEO 



PROJ. IREC IPROJjTAPE 



ECM 
SUIT. 



OPAQ! 



TV 



TRANR PROJ 'SYNC: HON PROJ;2C37 



ACM 
PROC. 

PVRDITRANR 



El Toro 

Beaufort 

Yuma 



0 
0 



Hiramar 
Oceana 
Dallas 
Andrews 



Iwakuni 0 



1 
0 



1 I 1 j 0 

1 ' 1 1 

i 

I 

oil 1 



0 

1 



0 1 1 



0 1 



1 1 



1 !-0 



0 
0 



27 
12 
11 



0 2k 
10 
75 
58 

1 12 



47 
29 
14 
47 



106 
84 
32. 
2 



12 I 0 
8 I 1 

10 3 

! 

I 

13 i 1 



62 4 



58 



53 



i 



0 23 



3 
3 



0 14 



4 

16 
6 



20 I 0 



-0 0 12 



3 0 8 



2 6 



25 



36 
26 



0 14 



0 I 2 



0 



1 



0 
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It is expected that the contracted development company will 
provide the required equipment . 

Facility requirements 'f'or development would include sufficient 
space for camera lay-up (8' x 10'), sound-proof recording (6' x 6*), 
and for graphic equipment (10' x 20'). Attendant tables, de^ks 
and chairs would also be required if on-site production was in- 
dicated. The facility should be sufficiently large to allow for 
heat dissipation from camera lights and to prevent unnecessary 
noise interference when recording. 

Facilities currently being used by the ISO team at MCAS , 
Yuma will provide sufficient space for materials production. 
These include two 12 x 10 offices for contractor personnel and 
two 11 X 11 offices for SME's. Graphics and photographic 
activities can be housed in a large 24 x 24 room. In addition, 
a 21 X 10 and a 12 x 11 room can be used as needed by production 
personnel- All rooms are well -lighted and have sufficient 
electrical power to support production efforts. A sound-proof 
recording facility is not available on-s^"te. This facility may 
be supplied by the narrator if he is a ^o 'essional with access 
to a recording facility, or in-service facilities at MCAS , El Toro 
and/or NAS , North Island may be used. A ^ark room is available 
at MCAS, Yuma. Its availability for contractor use could not 
be determined. 

IMPLEMENTATION REQUIREMENTS. 

It is estimated that a maximum of 30 instructors will be 
required to handle a student load of 60 pilots and 60 RIO's per 
year in a FRS . This estimate is derived from a maximun) require- 
ment of 16 flights per day with a different instructor for each 
flight, and 2 instructors per day for each of the three simulators, 
the learning center, and the Air Combat Maneuver Range (ACMR) . 
This totals to a maximum of 26 instructors available and allows 
approximately 15 percent for leave, TAD, etc., thus yielding 30 in- 
structors assigned. It is estimated that each instructor will 
be available four to six hours per day. 
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Other personnel requirements such as scheduling, counseling, 
proctor/monitoring, general administration, and evaluation can be 
met by the 30 assigned instructors. In addition, a minimum of 
two enlisted personnel will be required to distribute materials 
maintain student records, and service learning center equipment. 

The -primary equipment required" for implementation at each 
FRS will be audio-visual carrels in which the slide-tape lessons 
will be presented. A minimum of eight carrels will be ^^equired 
iit each FRS. Existing videotape equipment at the three FRS*s 
can be used for the videotape lessons. 

Each operational and reserve squadron provided wiuh course 
materials will require one functioning audio-visual carrel plus 
one spare. If feasible a complete, centralized learning center 
may be set up at locations having more than one squadron. 

Equipment costs to set up learning centers at the three 
FRS • s are summarized in Table D-9. The current costs are based 
on GSA prices; the projected costs include a 10 percent inflation factor 
to account for price increases prior to equipment purchase. The 
quantity of ST carrels required is eight per FRS with two 
additional cassette players and projectors per FRS included as 
on-1 i ne spares . 

It is recom.mended that two complete ST carrels be procured 
for each user squadron rather than one complete carrel plus 
on-line spares. One operational carrel will supply the training 
needs of each squadron. 

The learning center at each FRS should contain approximately 
350-400 square feet to accommodate the carrels, an ir:5^tructor 
station, and to allow free movement around the carrels. A separate, 
but preferably adjoining roon^ will house the lesso.i materials and 
will be used for storage and check in and out of materials. This 
room should be secure to allow for storage of classified materials. 
It should be a minimum of 100 square feet and contain 200 feet 
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TABLE D-9. COSTS OF LEARNING CENTER EQUIPMENT FOR THREE FSRs 



EQUIPMENT 

ST Carrel 

^owe TESS carrel 

Ecktographic AF-1 or 
AF-2 projector 

Telex C-140 Cassette 
tape player 

Rear projection niodule 

RAS Carrel 
Howe TESS carrel 
Kodak RA-960 projector 

Carrel (for „ lessons) 



CURRENT 
COST 

$142 
150 



319 



72 



142 

730 

142 



PROJECTED 

COST (w/inflation ) QUANTITY TOTAL 



$156 
165 



351 



79 



156 

803 

155 



24 (8/FRS) $ 3,744 
30 (lO/FRS) 4,950 



30 (10/FRS) 10,530 



24 ( 8/FRS) 1,896 



3 (1/FRS) 468 
6 (2/FRS) 4,818 



3 (1/FRS) 



468_ 

$26,874 
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of shelf space for materials' storage. A classroom capable of 
seating 10-15 students will be needed for lectures and periodic 
class meetings. Approximately eight briefing rooms will be required 
for 1 light l)ri(^f iri(js. Spcice lor i)r icf i rujs prior to simulator 
lessons will also be required. 

Facilities requirements at other squadrons will be minimum 
since there will be no more than two carrels and storage space 
for one set of materials per squadron. Approximately 150 square 
feet will be required. Materials can be stored in a secure 
cabinet in this space. 

A preliminary survey to determine the availability of 
facilities at the three FRS's indicated that the required space 
is avnilahlo. Specific rooms to ho used for the learning centers 
could not be identified. Based on the general characteristics of 
rooms that may be converted to learning centers, it is estimated 
that the following modifications will be required: (1) installation 
of carpeting in the 350-4C0 square foot learning center: (2) 
additional electrical power to provide the 4000-4500 watts required 
when oil carrels are in use: (3) installation of shelves or the 
procurement of cabinets to provide materials storage space. 
Classroom and briefing facilities were judged to be sufficient at 
all FRSs. 

FIVE YEAR EVALUATION AND MAINTENA^'^^- 

This phase of the ISD effort is concer.ied with program re- 
vision and maintenance requirements. After the training program 
has been implemented and evaluated, there will be first year 
revision requirements, curing the first year, many revisions will 
occur and be integrated into the training program. This requires 
more support personnel during the first year of a five year 
evaluation and maintenance effort than for the last four years. 

Total requirements for first year revisions and quality 
control activities were determined by multiplying the personnel, 
^ materials, and services directly tied to courseware development 
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and production by .35. Annual requirements for the next four 
years were determined by applying a factor of .20. 
RECOMMENDATIONS 

It is recognized that the TSRA results have indicated a 
costly development program for the F-4 Pilot and RIO courses. 
The estimated manhour requirements are believed to be very accurate 
for those personnel shown • The dollar cjinounts app»"Ox thm Led for* 
the develor^r.ient personnel are those that a noniial, well -ski lied 
professio 3l organization would charge. However, not all 
organizations utilize the same overhead rate, nor do all organi- 
zations have personnel skilled in segregated professions. That 
is, the Instructional Technologist may also be the Program 
Manager, the Production Manager may also be the proof reader, and 
the paste-up artist may be the graphics specialist or photographer. 
The most obvious cost area is the graphic specialist and photo- 
grapher in the slide- tape lesson area; however, they are required 
specialists. It is suggested that, in other areas, personnel who 
are mul ti - ta 1 ented be considered acceptable specialists and that 
the delineation of personnel categories by title not be utilized 
in the acquisition of a development team. In some cases, a SME 
could be utilized for production management or for paste-up work. 
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